Teaching Apparatus And Robot System

ABSTRACT

A teaching apparatus for a robot which moves a work object so that the work object is aligned at a predetermined position and attitude, includes a display section that displays a screen for setting the predetermined position and attitude, an operation section that allows the screen to be operated, and a computation section that computes the predetermined position and attitude, in which the screen includes a first screen for operating the robot so as to move the work object into an imaging range of an imaging device, and a second screen for moving the work object so that the work object is aligned at a target position and attitude, and in which the computation section computes the predetermined position and attitude by using a captured image obtained by the imaging device imaging the work object, and the target position and attitude.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/753,401, filed Jun. 29, 2015, which claims priority to JapanesePatent Application Nos. 2014-136315 filed Jul. 1, 2014, 2014-136316filed Jul. 1, 2014, and 2014-136317 filed Jul. 1, 2014, the entiredisclosures of which are expressly incorporated by reference herein intheir entireties.

BACKGROUND 1. Technical Field

The present invention relates to a teaching apparatus and a robotsystem.

2. Related Art

In the related art, a robot system which aligns a work object gripped(picked up) by a robot at a predetermined position and attitude has beenresearched and developed.

In relation thereto, a method of controlling a robot which can grip amember to be gripped with an appropriate gripping force to reliably gripthe member to be gripped without damaging the member to be grippedregardless of an appearance of the member to be gripped is known (referto JP-A-2013-132726).

In such a robot system, in order to accurately align the work object atthe predetermined position and attitude, it is necessary to detect(calculate) a position and an attitude of the picked-up work object anda deviation (offset) of the attitude relative to a reference attitude ofthe robot.

In relation thereto, there are a robot system in which an imagingsection detecting a position and an attitude of a work object gripped(picked up) by a robot is provided, and an image processing apparatusdetecting (calculating) a position and an attitude of the work object onthe basis of a captured image obtained by the imaging section isprovided separately from a control apparatus controlling the robot, anda method of capturing an image of the picked-up work object anddetecting a position and an attitude of the picked-up work object byusing the captured image (refer to JP-A-2012-230041).

However, in the method of the related art disclosed in JP-A-2013-132726,it is necessary to accurately teach the predetermined position andattitude to the robot in order to accurately align the work object atthe predetermined position and attitude. Also in other methods, it isimportant to accurately teach a predetermined position and attitude of awork object to a robot. However, in the methods of the related art, itis difficult to accurately teach the predetermined position andattitude.

In addition, in the robot system of the related art as disclosed inJP-A-2012-230041, the control apparatus receives an image processingresult from the image processing apparatus, and the control apparatusperforms computation regarding image processes such as calculation ofthe above-described offset or calculation of a position and an attitudeof a manipulator and a position and an attitude of a gripping portion,obtained in consideration of the offset.

For this reason, a program for performing such computation is requiredto be incorporated into a program which defines an operation of therobot, and thus a large load is put on a user associated with creationof the program. Since a created program differs for each controlapparatus, a large load is also put on the user associated with creationof the program. It is hard for the user to increase work efficiency dueto such a load when using the robot system of the related art. Further,not the image processing apparatus but the control apparatus performscomputation regarding image processes. As mentioned above, in a casewhere the control apparatus and the image processing apparatus sharecomputation regarding image processes, and the control apparatusperforms the computation regarding image processes, the robot may not beefficiently controlled compared with a case where the control apparatusperforms only control of the robot.

In such a method, in order to cause the robot to accurately align a workobject at a predetermined position and attitude, it is necessary toacquire an attitude of the work object in a pixel coordinate system, anda relationship (coordinate transform matrix) between the pixelcoordinate system and a robot coordinate system.

The relationship between the pixel coordinate system and the robotcoordinate system may be acquired through calibration of an imagingsection and the robot. On the other hand, an attitude of the work objectin the pixel coordinate system is acquired by the user creating aprogram, and thus this creation of the program puts a large burden onthe user.

SUMMARY

An advantage of some aspects of the invention is to provide a teachingapparatus and a robot system which can easily perform highly accurateteaching. Another advantage of some aspects of the invention is toprovide a teaching apparatus and a robot system capable of efficientlycontrolling a robot and reducing a burden on a user related to teaching.

One aspect of the invention is directed to a teaching apparatus for arobot which moves a work object so that the work object is aligned at apredetermined position and attitude, including a display section thatdisplays a screen for setting the predetermined position and attitude;an operation section that allows the screen to be operated; and acomputation section that computes the predetermined position andattitude, in which the screen includes a first screen for operating therobot so as to move the work object into an imaging range of an imagingdevice; and a second screen for moving the work object so that the workobject is aligned at a target position and attitude, and in which thecomputation section computes the predetermined position and attitude byusing a captured image obtained by the imaging device imaging the workobject, and the target position and attitude.

With this configuration, in the teaching apparatus, the first screen foroperating the robot so as to move the work object into the imaging rangeof the imaging device and the second screen for moving the work objectso that the work object is aligned at the target position and attitudeare operated by using the operation section, and the predeterminedposition and attitude are computed by using the captured image obtainedby the imaging device imaging the work object, and the target positionand attitude. Consequently, the teaching apparatus can easily performhighly accurate teaching.

In another aspect of the invention, in the teaching apparatus, either orboth of the first screen and the second screen may include a region inwhich the captured image obtained by the imaging device is displayed.

With this configuration, the teaching apparatus displays the capturedimage obtained by the imaging device on either or both of the firstscreen and the second screen. Consequently, the teaching apparatus canprovide a user with an environment in which the user moves the robotthrough a jog operation while checking the captured image.

In another aspect of the invention, in the teaching apparatus, either orboth of the first screen and the second screen may include a region inwhich a work procedure is displayed.

With this configuration, the teaching apparatus displays a workprocedure on either or both of the first screen and the second screen.Consequently, the teaching apparatus can reduce time and effort for theuser to read a manual and thus allows the user to perform efficientwork.

In another aspect of the invention, in the teaching apparatus, thecomputation section may calculate a relative position and attitudebetween the target position and attitude and the predetermined positionand attitude by using the captured image on the basis of an operationwhich is input via the first screen by using the operation section, andmay acquire the target position and attitude on the basis of anoperation which is input via the second screen by using the operationsection and may compute the predetermined position and attitude by usingthe calculated relative position and attitude and the acquired targetposition and attitude.

With this configuration, a relative position and attitude between thetarget position and attitude and the predetermined position and attitudeare calculated by using the captured image on the basis of an operationwhich is input via the first screen by using the operation section. Thetarget position and attitude are acquired on the basis of an operationwhich is input via the second screen by using the operation section andthe predetermined position and attitude are computed by using thecalculated relative position and attitude and the acquired targetposition and attitude. With this configuration, the teaching apparatuscan realize highly accurate work by calculating a relative position andattitude between a target position and attitude and a predeterminedposition and attitude for each piece of work.

One aspect of the invention is directed to a robot system including arobot that moves a work object so that the work object is aligned at apredetermined position and attitude; and a teaching apparatus thatteaches the predetermined position and attitude to the robot, in whichthe teaching apparatus includes a display section that displays a screenfor setting the predetermined position and attitude; an operationsection that allows the screen to be operated; and a computation sectionthat computes the predetermined position and attitude, in which thescreen includes a first screen for operating the robot so as to move thework object into an imaging range of an imaging device; and a secondscreen for moving the work object so that the work object is aligned ata target position and attitude, and in which the computation sectioncomputes the predetermined position and attitude by using a capturedimage obtained by the imaging device imaging the work object, and thetarget position and attitude.

With this configuration, in the robot system, the first screen foroperating the robot so as to move the work object into the imaging rangeof the imaging device and the second screen for moving the work objectso that the work object is aligned at the target position and attitudeare operated by using the operation section, and the predeterminedposition and attitude are computed by using the captured image obtainedby the imaging device imaging the work object, and the target positionand attitude. Consequently, the robot system can easily perform highlyaccurate teaching.

As described above, in the teaching apparatus and the robot system, thefirst screen for operating the robot so as to move the work object intothe imaging range of the imaging device and the second screen for movingthe work object so that the work object is aligned at the targetposition and attitude are operated by using the operation section, andthe predetermined position and attitude are computed by using thecaptured image obtained by the imaging device imaging the work object,and the target position and attitude. Consequently, the teachingapparatus and the robot system can easily perform highly accurateteaching.

One aspect of the invention is directed to a teaching apparatus for arobot which moves a work object, including a display section thatdisplays a screen for setting an attitude of the work object; anoperation section that allows the screen to be operated; and a detectionsection that detects the attitude of the work object, in which thescreen includes a first screen for displaying the work object placedwithin an imaging range of an imaging device; and a second screen forcausing the robot to grip the work object placed within the imagingrange of the imaging device, and in which the detection section detectsthe attitude of the work object by using a result of the robot grippingthe work object.

With this configuration, the teaching apparatus displays the firstscreen for displaying the work object placed within the imaging range ofthe imaging device and the second screen for causing the robot to gripthe work object placed within the imaging range of the imaging device,and detects the attitude of the work object by using a result of therobot gripping the work. Consequently, the teaching apparatus can reducea burden on a user related to teaching.

In another aspect of the invention, in the teaching apparatus, thedetection section may detect an attitude of the work object relative tothe imaging range as a reference attitude of the work object on thebasis of the work object placed within the imaging range of the imagingdevice, displayed on the first screen.

With this configuration, the teaching apparatus detects an attitude ofthe work object relative to the imaging range as a reference attitude ofthe work object on the basis of the work object placed within theimaging range of the imaging device, displayed on the first screen.Consequently, the teaching apparatus can teach the detected referenceattitude of the work object.

In another aspect of the invention, the teaching apparatus may furtherinclude a computation section that computes an attitude of another workobject placed within the imaging range by using the reference attitudeof the work object detected by the detection section and the result ofthe robot gripping the work object.

With this configuration, the teaching apparatus computes an attitude ofa work object which is newly placed within the imaging range by usingthe reference attitude of the work object detected by the detectionsection and the result of the robot gripping the work object.Consequently, the teaching apparatus can perform predetermined work onthe work object on the basis of an attitude of the work object computedby using the reference attitude of the work object and the result of therobot gripping the work object.

In another aspect of the invention, in the teaching apparatus, thesecond screen may include a region in which the captured image obtainedby the imaging device is displayed.

With this configuration, the teaching apparatus displays the capturedimage obtained by the imaging device on the second screen. Consequently,the teaching apparatus can provide a user with an environment in whichthe user moves the robot through an operation while checking thecaptured image.

In another aspect of the invention, in the teaching apparatus, either orboth of the first screen and the second screen may include a region inwhich a work procedure is displayed.

With this configuration, the teaching apparatus displays a workprocedure on either or both of the first screen and the second screen.Consequently, the teaching apparatus can reduce time and effort for theuser to read a manual and thus allows the user to perform efficientwork.

One aspect of the invention is directed to a robot system including arobot that moves a work object; and a teaching apparatus that teaches anattitude of the work object to the robot, in which the teachingapparatus includes a display section that displays a screen for settingan attitude of the work object; an operation section that allows thescreen to be operated; and a detection section that detects the attitudeof the work object, in which the screen includes a first screen fordisplaying the work object placed within an imaging range of an imagingdevice; and a second screen for causing the robot to grip the workobject placed within the imaging range of the imaging device, and inwhich the detection section detects the attitude of the work object byusing a result of the robot gripping the work object.

With this configuration, the robot system displays the first screen fordisplaying the work object placed within the imaging range of theimaging device and the second screen for causing the robot to grip thework object placed within the imaging range of the imaging device, anddetects the attitude of the work object by using a result of the robotgripping the work. Consequently, the robot system can reduce a burden ona user related to teaching.

As described above, the teaching apparatus and the robot system displaythe first screen for displaying the work object placed within theimaging range of the imaging device and the second screen for causingthe robot to grip the work object placed within the imaging range of theimaging device, and detect the attitude of the work object by using aresult of the robot gripping the work. Consequently, the teachingapparatus and the robot system can reduce a burden on a user related toteaching.

One aspect of the invention is directed to a robot system including arobot that moves a work object; an imaging device that images the workobject moved by the robot; a control apparatus that operates the robot;and an image processing apparatus that detects the work object moved bythe robot by using a captured image obtained by the imaging device, inwhich the image processing apparatus calculates a positionalrelationship between a first position of the robot and a second positionof the work object moved by the robot by using the captured image.

With this configuration, in the robot system, the image processingapparatus calculates a positional relationship between the firstposition of the robot and the second position of the work object movedby the robot by using the captured image. Consequently, the robot systemcan efficiently control the robot.

In another aspect of the invention, in the robot system, the robot mayinclude a hand and may move the work object gripped by the hand, and theimage processing apparatus may detect a predetermined position of thehand from the captured image as the first position.

With this configuration, the robot system detects a predeterminedposition of the hand from the captured image as the first position.Consequently, the robot system can calculate a positional relationshipbetween the predetermined position of the hand and the second positionof the work object.

In another aspect of the invention, in the robot system, the imageprocessing apparatus may detect the positional relationship between thefirst position of the robot and the second position of the work objectfrom the captured image through pattern matching.

With this configuration, the robot system detects the positionalrelationship between the first position of the robot and the secondposition of the work object from the captured image through patternmatching. Consequently, the robot system can calculate the positionalrelationship between the first position of the robot and the secondposition of the work object from the captured image through patternmatching.

As described above, in the robot system, the image processing apparatuscalculates the positional relationship between the first position of therobot and the second position of the work object moved by the robot byusing the captured image. Consequently, the robot system has highversatility related to replacement of the included apparatuses and canreduce time and effort required in the replacement.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is a diagram illustrating a configuration example of a robotsystem according to the present embodiment and an example of a state inwhich a robot performs predetermined work.

FIG. 2 is a flowchart illustrating an example of a flow of a processrelated to predetermined work performed by each device included in therobot system.

FIG. 3 is a diagram illustrating an example of a hardware configurationof a teaching apparatus.

FIG. 4 is a diagram illustrating an example of a functionalconfiguration of the teaching apparatus.

FIG. 5 is a flowchart illustrating an example of a flow of a process inwhich the teaching apparatus teaches a reference offset and target pointposition and attitude to a control apparatus.

FIG. 6 is a diagram illustrating an example of a setting screen.

FIG. 7 is a diagram illustrating an example of a first teaching screen.

FIG. 8 is a diagram illustrating an example of the first teaching screenand a jog screen.

FIG. 9 is a diagram illustrating an example of a positional relationshipbetween a work object and an imaging section at a predetermined imagingposition.

FIG. 10 is a diagram illustrating an example of a second teachingscreen.

FIG. 11 is a diagram illustrating an example of the second teachingscreen and a jog screen.

FIG. 12 is a diagram illustrating an example of a setting completionscreen.

FIG. 13 is a diagram illustrating an example of a configuration of arobot system according to the present embodiment and also illustrating astate in which a robot performs predetermined work.

FIG. 14 is a flowchart illustrating an example of a flow of a processrelated to predetermined work performed by each device included in therobot system.

FIG. 15 is a diagram illustrating an example of a functionalconfiguration of a teaching apparatus.

FIG. 16 is a flowchart illustrating an example of a flow of a process inwhich the teaching apparatus teaches reference information to a controlapparatus.

FIG. 17 is a diagram illustrating an example of a setting screen.

FIG. 18 is a diagram illustrating an example of a first teaching screen.

FIG. 19 is a diagram illustrating an example of an image captured by adetection unit on the basis of an operation which is received by aninput reception unit via the first teaching screen.

FIG. 20 is a diagram illustrating an example of a second teachingscreen.

FIG. 21 is a diagram illustrating an example of the second teachingscreen and a jog screen.

FIG. 22 is a diagram illustrating an example of a captured image whichis displayed in a region of the second teaching screen and is obtainedwhen a work object is gripped by a gripping portion through a jogoperation.

FIG. 23 is a diagram illustrating an example of a setting completionscreen.

FIG. 24 is a diagram illustrating an example of the setting screen afterteaching is completed.

FIG. 25 is a diagram illustrating a configuration of a robot systemaccording to the present embodiment.

FIG. 26 is a diagram illustrating an example of a hardware configurationof an image processing apparatus.

FIG. 27 is a diagram illustrating an example of a functionalconfiguration of the image processing apparatus.

FIG. 28 is a flowchart illustrating an example of a flow of a process inwhich the image processing apparatus calculates an offset.

FIGS. 29A and 29B illustrate an example of a state in which an imagingsection captures an image of an imaging range and an example of an imagecaptured by the imaging section.

DESCRIPTION OF EXEMPLARY EMBODIMENTS First Embodiment Outline

First, a summary of a robot system 1 according to the followingembodiment will be described, and then the embodiment will be describedin detail. FIG. 1 is a diagram illustrating an example of aconfiguration of the robot system 1 according to the present embodimentand illustrating an example of a state in which a robot 20 performspredetermined work.

The robot system 1 includes a teaching apparatus 5, an imaging section10, a single-arm robot 20 provided with a gripping portion HND (endeffector) and a manipulator MNP, and a control apparatus 30. In thepresent embodiment, the single-arm robot indicates a robot having asingle arm constituted by the gripping portion HND and the manipulatorMNP.

The robot system 1 teaches parameter information required for the robot20 to perform predetermined work to the control apparatus 30 whichcontrols the robot 20 before the robot 20 performs the predeterminedwork, by a user operating a graphical user interface (GUI) displayed onthe teaching apparatus 5. In the present embodiment, the predeterminedwork indicates that the robot 20 grips a work object O with the grippingportion HND and disposes the gripped work object O in a target area TA.

The robot system 1 illustrated in FIG. 1 causes the robot 20 to move thework object O to a predetermined imaging position and causes the imagingsection 10 to capture an image of the moved work object O when thegripped work object O is disposed in the target area TA in thispredetermined work. The robot system 1 controls the robot 20 so that thework object O gripped by the gripping portion HND is disposed in thetarget area TA on the basis of a captured image obtained by the imagingsection 10.

When the robot 20 is controlled so that the work object O gripped by thegripping portion HND is disposed in the target area TA on the basis ofthe captured image, the robot system 1 uses the parameter informationtaught by the teaching apparatus 5 before performing the predeterminedwork. More specifically, disposing the work object O in the target areaTA is to move the gripping portion HND so that a position and anattitude (hereinafter, referred to as a work object position andattitude) of a representative point OP which is set in the work object Oin advance match a position and an attitude (hereinafter, referred to astarget point position and attitude) of a target point TP of the targetarea TA. The target point position and attitude are an example of atarget position and attitude.

Here, the work object O is an object which can be gripped by the robot20, and is an industrial part such as a screw, a bolt, a gear, or atool, but is not limited thereto, and may be other objects as long asthe objects can be gripped by the robot 20. The representative point OPof the work object O is a point which is set in the control apparatus 30in advance, and is a feature point of the work object O and is one ofcorners of the work object O illustrated in FIG. 1 in this example. Thecorner is preferably a corner of a surface which is a bottom surfacewhen the work object O is disposed in the target area TA in order tofacilitate computation performed by the robot system 1. The work objectO is an example of a work object.

The target area TA is an area in which the work object O is disposed bythe robot 20, and is provided on a table TB in FIG. 1. The table TB is,for example, a stand on which the robot 20 disposes the work object O.The target point TP is a point in the target area TA in which the workobject O is disposed by the robot 20 and is a point which matches therepresentative point OP when the work object O is disposed in the targetarea TA at an accurate position and attitude.

Here, the robot system 1 illustrated in FIG. 1 moves a position and anattitude (hereinafter, referred to as a control point position andattitude) of a control point TCP of the gripping portion HND as aposition and an attitude of the gripping portion HND when the grippingportion HND of the robot 20 is moved. The control point TCP of thegripping portion HND indicates a position and an attitude of thegripping portion HND when the control apparatus 30 moves the grippingportion HND and the manipulator MNP of the robot 20, and indicates acentral point (a tool center point) of a flange provided at an end ofthe manipulator MNP at which the gripping portion HND is provided inthis example.

This control point position and attitude do not generally (except forspecial cases) match a work object position and attitude. For thisreason, the robot system 1 illustrated in FIG. 1 calculates a controlpoint position and attitude (hereinafter, referred to as a control pointtarget position and attitude) when the work object position and attitudematch the target point position and attitude on the basis of relativepositions and relative attitudes (hereinafter, referred to as an offset)between the control point position and attitude of the gripping portionHND and the work object position and attitude and the target pointposition and attitude, and can thus dispose the work object O in thetarget area TA in a desired state (that is, a state in which the workobject position and attitude match the target point position andattitude). The robot system 1 moves the gripping portion HND (that is,the control point TCP) so that the calculated control point targetposition and attitude match the control point position and attitude, andcan thus dispose the work object O in the target area TA in a desiredstate. The control point target position and attitude are an example ofa predetermined position and attitude.

The teaching apparatus 5 teaches an offset and a target point positionand attitude necessary to calculate the control point target positionand attitude to the control apparatus 30. In addition, the teachingapparatus 5 provides, to a user, an interface which allows the user toeasily teach the offset and the target point position and attitude tothe control apparatus 30 via a GUI realized by a dedicated application.Thus, the robot system 1 can easily perform highly accurate teachingwith the teaching apparatus 5. Hereinafter, the offset which is taughtto the control apparatus 30 by the teaching apparatus 5 is referred toas a reference offset.

Hereinafter, detailed description will be made of teaching performed bythe teaching apparatus 5. In the robot system 1, the teaching apparatus5 teaches the offset and the target point position and attitude to thecontrol apparatus 30, and then predetermined work is repeatedlyperformed on a plurality of work objects O. When the predetermined workis repeatedly performed, the robot system 1 causes the imaging section10 to capture an image of the work object O every time the predeterminedwork is performed (that is, every time a new work object O is gripped bythe gripping portion HND), and calculates an offset on the basis of thecaptured image. Hereinafter, the offset which is calculated in themiddle of the predetermined work being performed is referred to as acalculation offset.

The calculation offset is influenced by an error occurring when a newwork object O is gripped and thus does not generally match the referenceoffset. The robot system 1 calculates a control point target positionand attitude in a state in which the calculation offset is realized onthe basis of the reference offset, the target point position andattitude, and the calculation offset. Consequently, the robot system 1can cause the robot 20 to perform predetermined work with high accuracy.This is an effect obtained as a result of the teaching apparatus 5teaching the reference offset and the target point position and attitudeto the control apparatus 30 in the robot system 1. The predeterminedwork may be other pieces of work related to movement of the work objectO instead of the robot 20 disposing the gripped work object O in thetarget area TA.

Here, with reference to FIG. 2, a description will be made of a processrelated to predetermined work performed by each device included in therobot system 1. FIG. 2 is a flowchart illustrating an example of aprocess related to predetermined work performed by each device includedin the robot system 1. First, the control apparatus 30 causes the workobject O to be gripped (step S50). Next, the control apparatus 30 causesthe work object O to be moved to a predetermined imaging position (stepS51).

Next, the control apparatus 30 causes the imaging section 10 to capturean image via the teaching apparatus 5 (step S52). Next, the teachingapparatus 5 detects the work object O on the basis of the image capturedby the imaging section 10 (step S53). Next, the teaching apparatus 5calculates a calculation offset related to the work object O detected instep S53 on the basis of the image captured by the imaging section 10(step S54). The teaching apparatus 5 outputs the calculation offsetwhich has been calculated to the control apparatus 30.

Next, the control apparatus 30 calculates a control point targetposition and attitude on the basis of the calculation offset acquiredfrom the teaching apparatus 5, and the reference offset and the targetpoint position and attitude taught by the teaching apparatus 5 (stepS55). In addition, the teaching apparatus 5 may calculate a controlpoint target position and attitude, and the control apparatus 30 mayacquire the control point target position and attitude calculated by theteaching apparatus 5. Next, the control apparatus 30 moves the robot 20so that a control point position and attitude match the control pointtarget position and attitude calculated in step S55, and thus the workobject O is disposed in the target area TA (step S56).

Embodiment

Hereinafter, an embodiment of the invention will be described withreference to the drawings. As described above, the robot system 1according to the present embodiment illustrated in FIG. 1 includes theteaching apparatus 5, the imaging section 10, the robot 20, and thecontrol apparatus 30.

The teaching apparatus 5 is, for example, a notebook personal computer(PC), but may alternatively be a desktop PC, a tablet PC, a mobile phoneterminal, a multi-function mobile phone terminal (smartphone), apersonal digital assistant (PDA), or the like. The teaching apparatus 5is communicably connected to the imaging section 10 via a cable. Wiredcommunication using the cable is realized by a standard such as Ethernet(registered trademark) or a universal serial bus (USB). The teachingapparatus 5 may be connected to the imaging section 10 by using wirelesscommunication realized by a communication standard such as Wi-Fi(registered trademark).

The teaching apparatus 5 acquires a captured image from the imagingsection 10. A dedicated application is installed in the teachingapparatus 5 which displays a GUI for realizing the application. Theteaching apparatus 5 receives an operation from a user via the GUI, andcalculates a reference offset between a work object position andattitude of the work object O and a control point position and attitudeof the target point TP on the basis of the received operation and thecaptured image acquired from the imaging section 10.

If the user moves the gripping portion HND through a jog operation, theteaching apparatus 5 matches the work object position and attitude witha target point position and attitude, then receives an operation fromthe user via the GUI, and acquires information indicating a controlpoint target position and attitude from the control apparatus 30. Theteaching apparatus 5 calculates a target point position and attitude ofthe target area TA on the basis of the acquired control point targetposition and attitude and the calculated reference offset. Consequently,the teaching apparatus 5 can capture an image of a range including thetarget area TA with the imaging section, and can teach the target pointposition and attitude to the control apparatus 30 without performing aprocess of detecting the target point position and attitude on the basisof the captured image. A jog knob for performing a jog operation may beprovided in the teaching apparatus 5 and may be provided in the controlapparatus 30. In a case where the jog knob is provided in the teachingapparatus 5, the teaching apparatus 5 moves the robot 20 through a jogoperation via the control apparatus 30.

The teaching apparatus 5 outputs (teaches) information indicating thereference offset, and the target point position and attitude to thecontrol apparatus 30. The teaching apparatus 5 may output only theinformation indicating the reference offset to the control apparatus 30.In this case, the target point position and attitude are assumed to bestored in the control apparatus 30 in advance. Hereinafter, adescription will be made assuming that the teaching apparatus 5 outputsthe information indicating the reference offset and the target pointposition and attitude to the control apparatus 30.

When the robot system 1 causes the robot 20 to perform predeterminedwork, the teaching apparatus 5 causes the imaging section 10 to capturean image of a range including the work object O in response to a requestfrom the control apparatus 30. The teaching apparatus 5 calculates acalculation offset on the basis of the captured image. The teachingapparatus 5 calculates a calculation offset for each piece ofpredetermined work in response to a request from the control apparatus30, and calculates a control point target position and attitude for eachpiece of predetermined work on the basis of the reference offset, thetarget point position and attitude, and the calculation offset. In thiscase, the teaching apparatus 5 outputs information indicating thecalculated control point target position and attitude to the controlapparatus 30. As another configuration example, the teaching apparatus 5outputs information indicating the calculation offset to the controlapparatus 30 for each piece of predetermined work in response to arequest from the control apparatus 30. In this case, not the teachingapparatus 5 but the control apparatus 30 calculates a control pointtarget position and attitude.

The imaging section 10 is a camera including, for example, a chargecoupled device (CCD) or a complementary metal oxide semiconductor (CMOS)which is an imaging element converting collected light into an electricsignal. The imaging section 10 is a monocular camera, but may be formedof two or more cameras such as a stereo camera.

The imaging section 10 is communicably connected to the teachingapparatus 5 via a cable. Wired communication using the cable is realizedby a standard such as Ethernet (registered trademark) or a USB. Theimaging section 10 may be connected to the teaching apparatus 5 by usingwireless communication realized by a communication standard such asWi-Fi (registered trademark). The imaging section 10 and the teachingapparatus 5 may be formed not separately from each other but integrallywith each other.

The imaging section 10 is provided at a position where a range(hereinafter, referred to as an imaging range) including theabove-described control point TCP and the representative point OP of thework object O gripped by the gripping portion HND can be imaged when thegripping portion HND gripping the work object O is moved to apredetermined imaging position. In FIG. 1, the imaging section 10 isprovided at a position where an image of the above-described imagingrange can be captured vertically upward from below.

Instead of the configuration in which the imaging section 10 is providedat a position where an image of the above-described imaging range can becaptured vertically upward from below, there may be a configuration inwhich the imaging section 10 is provided at a position where an image ofthe imaging range can be captured in a horizontal direction, aconfiguration in which the imaging section 10 is provided at a positionwhere an image of the imaging range can be captured vertically downwardfrom above, and a configuration in which the imaging section 10 isprovided at a position where an image of the imaging range can becaptured from other directions. The imaging section 10 captures a movingimage as a captured image, but may alternatively capture a still image,or a still image and a moving image as a captured image. The imagingsection 10 is an example of an imaging device.

The robot 20 is a single-arm robot including, for example, the grippingportion HND, the manipulator MNP, and a plurality of actuator (notillustrated). The robot 20 may be a SCARA robot (horizontallyarticulated robot) or a dual-arm robot instead of the single-arm robot.The SCARA robot is a robot of which a manipulator is moved only in ahorizontal direction, and only a slide shaft at a front end of themanipulator is moved in a vertical direction. The dual-arm robot is arobot including two arms, each of which is constituted by the grippingportion HND and the manipulator MNP.

The arm of the robot 20 is of a six-axis vertically articulated type,and can perform an operation of a six-axial degree of freedom throughinterlocking operations among the support stand, the manipulator MNP,and the gripping portion HND using the actuators. The arm of the robot20 may operate in five or less degrees of freedom (five axes) or mayoperate in seven or more degrees of freedom (seven axes). Hereinafter, adescription will be made of an operation of the robot 20, performed bythe arm including the gripping portion HND and the manipulator MNP. Thegripping portion HND of the robot 20 is provided with claws which cangrip an object.

The robot 20 is communicably connected to the control apparatus 30 via,for example, a cable. Wired communication using the cable is realized bya standard such as Ethernet (registered trademark) or a USB. The robot20 may be connected to the control apparatus 30 by using wirelesscommunication realized by a communication standard such as Wi-Fi(registered trademark). In the robot system 1, the robot 20 isconfigured to be connected to the control apparatus 30 which is providedoutside the robot 20 as illustrated in FIG. 1, but the control apparatus30 may be built into the robot 20 instead of this configuration.

The robot 20 acquires a control signal generated by the controlapparatus 30, and moves the gripping portion HND and the manipulator MNPof the robot 20 on the basis of the acquired control signal.

In response to a request from the teaching apparatus 5, the controlapparatus 30 outputs, to the teaching apparatus 5, informationindicating a control point position and attitude (that is, a controlpoint target position and attitude) obtained when a state is realized inwhich the teaching apparatus 5 performs teaching and a work objectposition and attitude match a target point position and attitude througha user's jog operation. Then, the control apparatus 30 acquiresinformation indicating a reference offset and information indicating thetarget point position and attitude from the teaching apparatus 5.

When the robot system 1 causes the robot 20 to perform predeterminedwork, the control apparatus 30 causes the teaching apparatus 5 tocalculate a calculation offset on the basis of a captured image which isacquired by the teaching apparatus 5. The control apparatus 30 acquiresthe calculation offset which has been calculated from the teachingapparatus 5. The control apparatus 30 calculates a control point targetposition and attitude during the predetermined work on the basis of thereference offset, the target point position and attitude, and thecalculation offset.

Next, with reference to FIG. 3, a description will be made of a hardwareconfiguration of the teaching apparatus 5. FIG. 3 is a diagramillustrating an example of a hardware configuration of the teachingapparatus 5. The teaching apparatus 5 includes, for example, a centralprocessing unit (CPU) 51, a storage section 52, an input receptionsection 53, a communication section 54, and a display section 55, andperforms communication with the control apparatus 30 via thecommunication section 54. These constituent elements are communicablyconnected to each other via a bus Bus. The CPU 51 executes variousprograms stored in the storage section 52.

The storage section 52 includes, for example, a hard disk drive (HDD), asolid state drive (SSD), an electrically erasable programmable read-onlymemory (EEPROM), a read only memory (ROM), and a random access memory(RAM), and stores various information pieces or images processed by theteaching apparatus 5, and programs. Instead of being built into theteaching apparatus 5, the storage section 52 may be an externallyattached storage device which is connected to a digital input and outputport such as a USB.

The input reception section 53 is, for example, a keyboard, a mouse, atouch pad, and other input devices. The input reception section 53 maybe configured of a touch panel so as to function as a display section.The input reception section 53 is an example of an operation section.

The communication section 54 includes, for example, a digital input andoutput port such as a USB, or an Ethernet port.

The display section 55 is, for example, a liquid crystal display panelor an organic electroluminescent display panel.

Next, with reference to FIG. 4, a functional configuration of theteaching apparatus 5 will be described. FIG. 4 is a diagram illustratingan example of a functional configuration of the teaching apparatus 5.The teaching apparatus 5 includes the input reception section 53, thecommunication section 54, the display section 55, an image acquisitionsection 56, and a control section 60. Some or all function unitsincluded in the control section 60 are realized by, for example, the CPU51 executing the various programs stored in the storage section 52. Inaddition, some or all of the function units may be hardware functionunits such as a large scale integration (LSI) circuit or an applicationspecific integrated circuit (ASIC).

The image acquisition section 56 acquires a captured image from theimaging section 10. The image acquisition section 56 outputs theacquired captured image to the control section 60.

The control section 60 includes a display control unit 61, a computationunit 62, a communication control unit 67, and an imaging control unit68. The control section 60 controls the entire teaching apparatus 5.

The display control unit 61 generates a GUI for teaching a referenceoffset and a target point position and attitude to the control apparatus30 on the basis of the acquired captured image, and displays thegenerated GUI. More specifically, the display control unit 61 generatesa screen (GUI) for calculating a reference offset as a first teachingscreen on the basis of a user's operation which is input by using theinput reception section 53. The display control unit 61 controls thedisplay section 55 so as to display the generated first teaching screen.The display control unit 61 generates a screen for calculating a targetpoint position and attitude as a second teaching screen. The displaycontrol unit 61 controls the display section 55 to display the secondteaching screen on the basis of a user's operation which is input byusing the input reception section 53.

The computation unit 62 includes a first calculation portion 63, asecond calculation portion 64, and a third calculation portion 65.

The first calculation portion 63 calculates (computes) a referenceoffset or a calculation offset between a control point position andattitude and a work object position and attitude of the work object O onthe basis of the captured image acquired from the image acquisitionsection 56. When the reference offset is calculated, the firstcalculation portion 63 stores the calculated reference offset in thestorage section 52.

The second calculation portion 64 acquires information indicating acontrol point target position and attitude from the control apparatus 30via the communication section 54 controlled by the communication controlunit 67. The second calculation portion 64 reads the reference offsetfrom the storage section 52. The second calculation portion calculates(computes) a target point position and attitude on the basis of thecontrol point target position and attitude acquired from the controlapparatus 30 and the reference offset (that is, the reference offsetcalculated by the first calculation portion 63) read from the storagesection 52.

The third calculation portion 65 reads the reference offset from thestorage section 52. The third calculation portion 65 calculates(computes) a control point target position and attitude obtained whenthe robot system 1 causes the robot 20 to perform predetermined work onthe basis of the reference offset (that is, the reference offsetcalculated by the first calculation portion 63) read from the storagesection 52, the calculation offset calculated by the first calculationportion 63, and the target point position and attitude calculated by thesecond calculation portion 64.

The communication control unit 67 controls the communication section 54so that the reference offset calculated by the first calculation portion63 is output to the control apparatus 30 on the basis of a user'soperation which is input by using the input reception section 53 via thefirst teaching screen which is displayed on the display section 55 bythe display control unit 61. The communication control unit 67 controlsthe communication section 54 so that the target point position andattitude calculated by the second calculation portion 64 is output tothe control apparatus 30 on the basis of a user's operation which isinput by using the input reception section 53 via the second teachingscreen which is displayed on the display section 55 by the displaycontrol unit 61. The communication control unit 67 causes thecalculation offset calculated by the first calculation portion 63 or thecontrol point target position and attitude calculated by the thirdcalculation portion 65 to be output to the control apparatus 30 inresponse to a request from the control apparatus 30.

The imaging control unit 68 controls the imaging section 10 to capturean image of an imaging possible range.

Hereinafter, with reference to FIG. 5, a description will be made of aprocess in which the teaching apparatus 5 teaches the reference offsetand the target point position and attitude to the control apparatus 30.FIG. 5 is a flowchart illustrating an example of a flow of a process inwhich the teaching apparatus 5 teaches the reference offset and thetarget point position and attitude to the control apparatus 30. First,the display control unit 61 displays a setting screen on the basis of auser's operation which is input by using the input reception section 53(step S80). Next, the control section 60 of the teaching apparatus 5receives a user's operation which is input via the setting screen byusing the input reception section 53 (step S90). Here, with reference toFIG. 6, the setting screen will be described. FIG. 6 is a diagramillustrating an example of the setting screen.

The display control unit 61 acquires a captured image obtained by theimaging section 10 from the image acquisition section 56, and displaysthe acquired captured image in a region V1. The display control unit 61displays a property column Q1 on the setting screen illustrated in FIG.6. The property column Q1 is a column which displays a list of itemswhich can be taught to the control apparatus 30 by the teachingapparatus 5 and is a column in which a name of each item and a set valueof the item are displayed in correlation with each other.

Here, if the user presses (clicks on) Click to calibrate B0 of a setvalue column B0 correlated with the property CalRobotPlacePos (anexample of the item) by using the input reception section 53, thedisplay control unit 61 displays the first teaching screen described instep S100.

The display control unit 61 displays a result column Q2 on the settingscreen illustrated in FIG. 6. The result column Q2 is a column whichdisplays various information pieces which have already been detected andare taught to the control apparatus 30 on the basis of the capturedimage obtained by the imaging section 10, and is a column which displaysa name of each item and a detection result for each item in correlationwith each other.

Hereinafter, a description will be made assuming that the user pressesthe button B0 illustrated in FIG. 6 by using the input reception section53 in step S90. Next, the display control unit 61 generates the firstteaching screen on the basis of the user's operation (that is, anoperation of pressing the button B0) which is input by using the inputreception section 53 in step S90. The display control unit 61 controlsthe display section 55 to display the generated first teaching screen(step S100).

Next, the control section 60 of the teaching apparatus 5 receives auser's operation which is input via the first teaching screen by theinput reception section 53 (step S110). Here, with reference to FIG. 7,the first teaching screen will be described. FIG. 7 is a diagramillustrating an example of the first teaching screen. The first teachingscreen UI1 displays an explanation image E1 for showing proceduresperformed when the control apparatus 30 is taught via the first teachingscreen UI1. If the user follows the explanation shown by the explanationimage E1, the user can easily perform an operation for teaching thecontrol apparatus 30 without reading a manual.

On the first teaching screen UI1, in addition to the explanation imageE1, a captured image obtained by the imaging section 10 is displayed ina region P1. The display control unit 61 acquires the captured imageobtained by the imaging section 10 from the image acquisition section 56and displays the acquired captured image in the region P1. If the firstteaching screen UI1 is displayed on the display section 55, the userpresses a button B1 which is displayed on the first teaching screen UI1.Consequently, the user can perform an operation for teaching the controlapparatus 30 according to the procedures having the content shown by theexplanation image E1.

If the user presses the button B1, the display control unit 61 displaysa jog screen UI1-1 as a sub-screen of the first teaching screen UI1 asillustrated in FIG. 8. FIG. 8 is a diagram illustrating an example ofthe first teaching screen UI1 and the jog screen UI1-1. A plurality ofbuttons for moving the control point of the gripping portion HND of therobot 20 to a position desired by the user are displayed in a region JGon the jog screen UI1-1. For example, a +X button is a button for movingthe control point of the gripping portion HND in a positive direction ofthe X axis, and a −X button is a button for moving the control point ofthe gripping portion HND in a negative direction of the X axis.

In addition, a +Y button is a button for moving the control point of thegripping portion HND in a positive direction of the Y axis, and a −Ybutton is a button for moving the control point of the gripping portionHND in a negative direction of the Y axis. A +Z button is a button formoving the control point of the gripping portion HND in a positivedirection of the Z axis, and a −Z button is a button for moving thecontrol point of the gripping portion HND in a negative direction of theZ axis.

A +U button is a button for rotating the control point of the grippingportion HND in a positive direction about the X axis, and a −U button isa button for rotating the control point of the gripping portion HND in anegative direction about the X axis. A +V button is a button forrotating the control point of the gripping portion HND in a positivedirection about the Y axis, and a −V button is a button for rotating thecontrol point of the gripping portion HND in a negative direction aboutthe Y axis. A +W button is a button for rotating the control point ofthe gripping portion HND in a positive direction about the Z axis, and a−W button is a button for rotating the control point of the grippingportion HND in a negative direction about the Z axis.

The display control unit 61 displays, for example, the presentcoordinates or the like in a robot coordinate system of the controlpoint on the jog screen UI1-1. If the jog screen UI1-1 illustrated inFIG. 8 is displayed on the display section 55, the user causes the robot20 to grip the work object O with the gripping portion HND according tothe procedures shown by the explanation image E1 displayed on the firstteaching screen UI1. The user causes the gripping portion HND of therobot 20 gripping the work object O to be moved to a predeterminedimaging position. The user causes the robot 20 to grip the work object Owith the gripping portion HND and to move the work object O to thepredetermined imaging position through a jog operation via the jogscreen UI1-1.

The user can check whether or not the work object O has been moved tothe predetermined imaging position while viewing the captured imagedisplayed in the region P1. Here, the predetermined imaging position isa position where the control point TCP of the gripping portion HND andthe representative point OP of the work object O gripped by the grippingportion HND can be imaged by the imaging section 10.

Here, with reference to FIG. 9, the predetermined imaging position willbe described. FIG. 9 is a diagram illustrating an example of apositional relationship between the work object O and the imagingsection 10 at the predetermined imaging position. A state of the workobject O gripped by the gripping portion HND in FIG. 9 is a state inwhich the work object O has been moved to the imaging position through ajog operation performed by the user. After the work object O has beenmoved to the imaging position, the imaging section 10 captures an imageof an imaging possible range in a direction C illustrated in FIG. 9. Thedirection C is a direction perpendicular to a face M of the work objectO gripped by the gripping portion HND. The face M is a face whichopposes the flange F at which the gripping portion HND is provided whenthe work object O is gripped by the gripping portion HND. Therefore, theimaging section 10 captures an image of the imaging range in a state inwhich the face M is parallel to a plane of the imaging element.

The face M being made parallel to the plane of the imaging element isaimed at detecting an offset which is obtained when the work object O isdisposed in the target area TA at an accurate position and attitude andis an offset between a control point position and attitude and arepresentative point position and attitude on a target area face (thatis, on a plane of the table TB) when the control point TCP and therepresentative point OP are projected onto the target area face in thetarget area TA. As mentioned above, an offset is treated as an offset onthe target area face, and thus computation cost of the teachingapparatus 5 or the control apparatus 30 can be minimized. Such offsettreatment is only an example, and other offset treatment methods may beused.

The captured image of the work object O which has already been moved tothe predetermined imaging position is displayed in the region P1illustrated in FIG. 8. After the user causes the work object O to bemoved to the predetermined imaging position, the user presses a teachbutton B2 on the jog screen UI1-1 illustrated in FIG. 8. If the userpresses the teach button B2, the first calculation portion 63 capturesan image which is being displayed on the first teaching screen UI1 whenthe teach button B2 is pressed, and detects a control point position andattitude and a work object position and attitude on the basis of animage processing method such as pattern matching on the basis of thecaptured image.

For example, the first calculation portion 63 detects a position of thecontrol point TCP and an attitude of the control point TCP by performingpattern matching on the basis of shapes of the gripping portion HND andthe manipulator MNP included in the captured image. In addition, thefirst calculation portion 63 detects a position of the representativepoint OP and an attitude of the representative point OP by performingthe pattern matching on the basis of a shape of the work object Oincluded in the captured image. The first teaching screen UI1 and thejog screen UI1-1 (that is, the screens illustrated in FIG. 8) are anexample of a first screen.

After the user presses the teach button B2 and the control pointposition and attitude and the work object position and attitude aredetected in step S110, the first calculation portion 63 calculates areference offset on the basis of the detected work object position andattitude and the target point position and attitude (step S120). Here,after the reference offset is calculated, the display control unit 61controls the display section 55 to erase the first teaching screen UI1and the jog screen UI1-1. However, alternatively, the first teachingscreen may not be erased until the user inputs an operation for erasingthe first teaching screen UI1 and the jog screen UI1-1 by using theinput reception section 53.

Next, the communication control unit 67 controls the communicationsection 54 to output (teach) information indicating the reference offsetcalculated in step S120 to the control apparatus 30 (step S130). Next,the display control unit 61 generates the second teaching screen on thebasis of the user's operation which is input by using the inputreception section 53. The display control unit 61 controls the displaysection 55 to display the generated second teaching screen (step S140).

Instead of the configuration in which the display control unit 61generates the second teaching screen on the basis of the user'soperation which is input by using the input reception section 53, thedisplay control unit 61 may generate the second teaching screen when theinformation indicating the reference offset is output to the controlapparatus 30 in step S130, or may generate the second teaching screenwith erasure of the first teaching screen as a trigger.

Next, the control section 60 of the teaching apparatus 5 receives auser's operation which is input via the second teaching screen by usingthe input reception section 53 (step S150). Next, the second calculationportion 64 calculates a target point position and attitude on the basisof the received operation (step S160). Here, with reference to FIG. 10,a description will be made of the second teaching screen and processesin steps S140 to S160 related to the second teaching screen. FIG. 10 isa diagram illustrating an example of the second teaching screen.

The second teaching screen UI2 displays an explanation image E2 forshowing procedures performed when the control apparatus 30 is taught viathe second teaching screen UI2. If the user follows the explanationshown by the explanation image E2, the user can easily perform anoperation for teaching the control apparatus 30 without reading amanual.

On the second teaching screen UI2, in addition to the explanation imageE2, a captured image obtained by the imaging section 10 is displayed ina region P2. The display control unit 61 acquires the captured imageobtained by the imaging section 10 from the image acquisition section 56and displays the acquired captured image in the region P2. If the secondteaching screen UI2 is displayed on the display section 55, the userpresses a button B3 which is displayed on the second teaching screenUI2. Consequently, the user can perform an operation for teaching thecontrol apparatus 30 according to the procedures having the contentshown by the explanation image E2.

If the user presses the teach button B2, the display control unit 61displays a jog screen UI2-1 as a sub-screen of the second teachingscreen UI2 as illustrated in FIG. 11. FIG. 11 is a diagram illustratingan example of the second teaching screen UI2 and the jog screen UI2-1.The jog screen UI2-1 is the same as the jog screen UI1-1 and thusdescription thereof will be omitted.

If the jog screen UI2-1 illustrated in FIG. 11 is displayed on thedisplay section 55, the work object O gripped by the gripping portionHND is moved to the target area TA through a jog operation via the jogscreen UI2-1 according to the procedures shown by the explanation imageE2 displayed on the second teaching screen UI2. The user matches thework object position and attitude with the target point position andattitude.

After the user matches the work object position and attitude with thetarget point position and attitude through a jog operation via the jogscreen UI2-1, the user presses a teach button B4 on the jog screen UI2-1illustrated in FIG. 11. If the user presses the teach button B4, thesecond calculation portion 64 acquires information indicating thecontrol point position and attitude when the teach button B4 is pressed,from the control apparatus 30 as information indicating a control pointtarget position and attitude. Then, the second calculation portion 64calculates the target point position and attitude on the basis of theacquired control point target position and attitude and the referenceoffset calculated by the first calculation portion 63.

As mentioned above, the teaching apparatus 5 allows the user to move thework object O through a jog operation via the jog screen UI2-1 so as tobe aligned at a target position where the work object O is actuallydisposed and at a target attitude, and calculates a target pointposition and attitude on the basis of the control point TCP and theabove-described reference offset at the position and the attitude.Therefore, it is possible to teach the target point position andattitude to the control apparatus 30 with higher accuracy than, forexample, in a case where the target point position and attitude aredetected on the basis of a captured image. As a result, since thecontrol apparatus 30 can calculate a control point target position andattitude with high accuracy when performing predetermined work, therobot 20 can perform highly accurate work.

On the second teaching screen UI2, in addition to the explanation imageE2, a captured image in the first teaching screen UI1 is displayed inthe region P2. The display control unit 61 may acquire a captured imageobtained by the imaging section 10 from the image acquisition section 56and may display the acquired captured image in the region P2. The secondteaching screen UI2 and the jog screen UI2-1 (that is, the screensillustrated in FIG. 11) are an example of a second screen.

Next, the communication control unit 67 controls the communicationsection 54 to output (teach) information indicating the target pointposition and attitude calculated in step S160 to the control apparatus30 (step S170). Instead of the configuration in which the informationindicating the reference offset is output to the control apparatus 30 instep S130, the communication control unit 67 may perform control so thatthe information indicating the reference offset is output to the controlapparatus 30 along with the information indicating the target pointposition and attitude in step S170.

Next, the display control unit 61 generates a setting completion screen.The display control unit 61 controls the display section 55 to displaythe generated setting completion screen (step S180). Here, withreference to FIG. 12, the setting completion screen will be described.FIG. 12 is a diagram illustrating an example of the setting completionscreen. The display control unit 61 displays an image or explanation fornotifying that setting has been completed in a column E3 on the settingcompletion screen. As illustrated in FIG. 12, the display control unit61 may display a captured image P3 obtained by the imaging section 10 onthe setting completion screen but may not display the captured image.The display control unit 61 displays a completion button B5 on thesetting completion screen. If the user presses the completion button B5,the display control unit 61 erases the setting completion screen.

The robot system 1 according to the present embodiment calculates areference offset, a calculation offset, a target point position andattitude, and a control point target position and attitude by using thecomputation unit 62, but, alternatively, may calculate some or all ofthe reference offset, the calculation offset, the target point positionand attitude, and the control point target position and attitude byusing a separate apparatus. In this case, the teaching apparatus 5requests the apparatus to calculate some or all of the reference offset,the calculation offset, the target point position and attitude, and thecontrol point target position and attitude via the control apparatus 30.

As described above, in the robot system 1 according to the presentembodiment, the user operates the first teaching screen UI1 foroperating the robot so as to move the work object O into an imagingpossible range of the imaging section 10 and the second teaching screenUI2 for moving the work object O so as to be aligned at a target pointposition and attitude by using the input reception section 53, and thusa work point target position and attitude are calculated by using acaptured image obtained by the imaging section 10 capturing an image ofthe work object O and the target point position and attitude.Consequently, the robot system 1 can easily perform highly accurateteaching.

In the robot system 1, the first teaching screen UI1 includes the regionP1 in which a captured image obtained by the imaging section 10 isdisplayed, and the second teaching screen UI2 includes the region P2 inwhich a captured image which is obtained by the imaging section 10 andis displayed on the first teaching screen UI1 is displayed.Consequently, the robot system 1 can provide the user with anenvironment in which the user moves the robot 20 through a jog operationwhile checking the captured image.

In the robot system 1, the first teaching screen UI1 includes the regionin which work procedures are displayed and the second teaching screenU12 includes the region in which work procedures are displayed.Consequently, the robot system 1 can reduce time and effort for the userto read a manual and thus allows the user to perform efficient work.

The robot system 1 calculates a reference offset between a target pointposition and attitude and a control point target position and attitudeby using a captured image on the basis of an operation which is inputvia the first teaching screen UI1 by using the input reception section53, acquires the target point position and attitude from the controlapparatus 30 on the basis of an operation which is input via the secondteaching screen U12 by using the input reception section 53, andcalculates the control point target position and attitude by using thecalculated reference offset and the acquired target point position andattitude. Consequently, the robot system 1 can realize highly accuratework by calculating a calculation offset between the target pointposition and attitude and the control point target position and attitudefor piece of work.

Second Embodiment Outline

First, a summary of a robot system 101 according to the followingembodiment will be described, and then the embodiment will be describedin detail. In the present embodiment, the same constituent elements asin the first embodiment are given the same reference numerals, anddescription thereof will be omitted or will be made briefly.

FIG. 13 is a diagram illustrating an example of a configuration of therobot system 101 according to the present embodiment and alsoillustrating an example of a state in which a robot 20 performspredetermined work. The robot system 101 includes a teaching apparatus105, an imaging section 10, a single-arm robot 20 provided with agripping portion HND (end effector) and a manipulator MNP, and a controlapparatus 130. In the present embodiment, the single-arm robot indicatesa robot having a single arm constituted by the gripping portion HND andthe manipulator MNP.

Here, in the present embodiment, the predetermined work indicates thatthe robot 20 grips a work object O placed on a table TB with thegripping portion HND and disposes the gripped work object O at anarrangement position X which is registered in advance. The table TB is astand on which the work object O is placed. The work object O is anobject which can be gripped by the robot 20, and is an industrial partsuch as a nut, a bolt, a gear, or a tool, but is not limited thereto,and may be other objects as long as the objects can be gripped by therobot 20. In FIG. 13, the work object O is illustrated as a rectangularparallelepiped object.

In the robot system 101, the teaching apparatus 105 teaches referenceinformation required for the robot 20 to perform predetermined work tothe control apparatus 130 before the robot 20 performs the predeterminedwork. In the robot system 101, when the teaching apparatus 105 teachesthe reference information to the control apparatus 130, an operation isreceived from the user via a graphical user interface (GUI) displayed onthe teaching apparatus 105 and thus the reference information is taughtto the control apparatus 130.

After the teaching apparatus 105 teaches the reference information tothe control apparatus 130, the robot system 101 causes the robot 20 toperform the predetermined work. More specifically, the robot system 101causes the imaging section 10 to capture an image of a range includingthe work object O. The robot system 101 detects (calculates) a positionand an attitude of the work object O on the basis of the captured imageobtained by the imaging section 10. Hereinafter, for convenience ofdescription, a position and an attitude of the work object O arereferred to as a work object position and attitude.

The robot system 101 detects a position and an attitude of arepresentative point OP which is set in the work object O in advancefrom the captured image obtained by the imaging section 10 as the workobject position and attitude. The representative point OP of the workobject O is a point which is set in the control apparatus 130 inadvance, and is a feature point of the work object O and is one ofcorners of the work object O illustrated in FIG. 13 in this example. Thecorner is preferably a corner of a surface which is a bottom surfacewhen the work object O is disposed at the arrangement position X inorder to facilitate computation performed by the robot system 101. Thework object O is an example of the work object.

The robot system 101 calculates a position and an attitude of thegripping portion HND right before the gripping portion HND of the robot20 grips the work object O on the basis of the work object position andattitude detected from the captured image and the reference informationtaught by the teaching apparatus 105, and moves the gripping portion HNDof the robot 20 so as to be aligned at the calculated position andattitude.

Here, the robot system 101 illustrated in FIG. 13 moves a position andan attitude (hereinafter, referred to as a control point position andattitude) of a control point TCP of the gripping portion HND as aposition and an attitude of the gripping portion HND when the grippingportion HND of the robot 20 is moved. The control point TCP of thegripping portion HND indicates a position and an attitude of thegripping portion HND when the control apparatus 130 moves the grippingportion HND and the manipulator MNP of the robot 20, and indicates acentral point (a tool center point) of a flange provided at an end ofthe manipulator MNP at which the gripping portion HND is provided inthis example. Hereinafter, for convenience of description, a positionand an attitude of the gripping portion HND right before the work objectO is gripped are referred to as a control point target position andattitude.

After the gripping portion HND of the robot 20 is moved so as to matchthe control point position and attitude of the gripping portion HND withthe control point target position and attitude (that is, right beforethe work object O is gripped), the robot system 101 causes the grippingportion HND of the robot 20 to grip the work object O. The robot system101 causes the robot 20 to dispose the work object O gripped by thegripping portion HND at the arrangement position X.

Hereinafter, detailed description will be made of teaching performed bythe teaching apparatus 105. In the robot system 101, the teachingapparatus 105 teaches the reference information to the control apparatus130, and then predetermined work is repeatedly performed on a pluralityof work objects O. When the predetermined work is repeatedly performed,the robot system 101 causes the imaging section 10 to capture an imageof the work object O every time the predetermined work is performed(that is, every time a new work object O is placed on the table TB, orfor each newly detected work object O), calculates a work objectposition and attitude on the basis of the captured image, and calculatesa new control point target position and attitude for gripping the workobject O on the basis of the calculated work object position andattitude and the reference information taught by the teaching apparatus105.

Here, with reference to FIG. 14, a description will be made of a processrelated to predetermined work performed by each device included in therobot system 101. FIG. 14 is a flowchart illustrating an example of aprocess related to predetermined work performed by each device includedin the robot system 101. First, the control apparatus 130 causes theimaging section 10 to capture an image via the teaching apparatus 105(step S250). Next, the teaching apparatus 105 detects the work object Oon the basis of the captured image obtained by the imaging section 10(step S251). Next, the control apparatus 130 calculates a position andan attitude of the detected work object O (that is, a work objectposition and attitude) (step S252). The teaching apparatus 105 outputsthe calculated work object position and attitude to the controlapparatus 130.

Next, the control apparatus 130 acquires the work object position andattitude from the teaching apparatus 105 (step S253). Next, the controlapparatus 130 calculates a control point target position and attitude onthe basis of the work object position and attitude acquired in step S253and the reference information taught by the teaching apparatus 105 (stepS254). Next, the control apparatus 130 causes the robot 20 to grip thework object O on the basis of the calculated control point targetposition and attitude (step S255). Next, the control apparatus 130controls the robot 20 so that the work object O is disposed at thearrangement position X (step S256).

Embodiment

Hereinafter, an embodiment of the invention will be described withreference to the drawings. As described above, the robot system 101according to the present embodiment illustrated in FIG. 13 includes theteaching apparatus 105, the imaging section 10, the robot 20, and thecontrol apparatus 130.

The teaching apparatus 105 is, for example, a notebook personal computer(PC), but may alternatively be a desktop PC, a tablet PC, a mobile phoneterminal, a multi-function mobile phone terminal (smartphone), apersonal digital assistant (PDA), or the like. The teaching apparatus105 is communicably connected to the imaging section 10 via a cable.Wired communication using the cable is realized by a standard such asEthernet (registered trademark) or a universal serial bus (USB). Theteaching apparatus 105 may be connected to the imaging section 10 byusing wireless communication realized by a communication standard suchas Wi-Fi (registered trademark).

The teaching apparatus 105 acquires a captured image from the imagingsection 10. A dedicated application is installed in the teachingapparatus 105 which displays a GUI for realizing the application. Theteaching apparatus 105 receives an operation from a user via the GUI,and detects a work object position and attitude of the work object O asa work object reference position and attitude on the basis of thereceived operation and the captured image acquired from the imagingsection 10.

If the user moves the gripping portion HND through a jog operation, theteaching apparatus 105 realizes a state which is desired by the user andis obtained right before the gripping portion HND grips the work objectO, then receives an operation from the user via the GUI, and acquiresinformation indicating a control point position and attitude from thecontrol apparatus 130 as information indicating a control point targetposition and attitude. The teaching apparatus 105 calculates an offsetbetween the work object reference position and attitude and the controlpoint target position and attitude as a reference offset based on theacquired control point target position and attitude and the detectedwork object reference position and attitude. The teaching apparatus 105outputs (teaches) the calculated reference offset and informationindicating the work object reference position and attitude to thecontrol apparatus 130 as reference information.

A jog knob for performing a jog operation may be provided in theteaching apparatus 105 and may be provided in the control apparatus 130.In a case where the jog knob is provided in the teaching apparatus 105,the teaching apparatus 105 moves the robot 20 through a jog operationvia the control apparatus 130. The teaching apparatus 105 may output theinformation only indicating the work object reference position andattitude to the control apparatus 130 as the reference information.

In this case, the control apparatus 130 calculates a reference offset onthe basis of the acquired (taught) work object reference position andattitude and a control point target position and attitude which arestored therein, and stores again the work object reference position andattitude as reference information along with the calculated referenceoffset. Hereinafter, a description will be made assuming that theteaching apparatus 105 outputs the information indicating the workobject reference position and attitude and the reference offset to thecontrol apparatus 130 as reference information.

When the robot system 101 causes the robot 20 to perform predeterminedwork, the teaching apparatus 105 causes the imaging section 10 tocapture an image of a range including the work object O in response to arequest from the control apparatus 130. The teaching apparatus 105detects a work object position and attitude on the basis of the capturedimage. The teaching apparatus 105 detects the work object position andattitude for each piece of the predetermined work in response to arequest from the control apparatus 130. The teaching apparatus 105outputs information indicating the detected work object position andattitude to the control apparatus 130 as detection information.

Instead of the configuration of outputting information indicating thedetected work object position and attitude to the control apparatus 130as detection information, the teaching apparatus 105 may acquirereference information from the control apparatus 130, and may calculatea control point target position and attitude on the basis of theacquired reference information and the detected work object position andattitude. In this case, the teaching apparatus 105 outputs informationindicating the calculated control point target position and attitude tothe control apparatus 130.

The imaging section 10 is a camera including, for example, a chargecoupled device (CCD) or a complementary metal oxide semiconductor (CMOS)which is an imaging element converting collected light into an electricsignal. The imaging section 10 is a monocular camera, but may be formedof two or more cameras such as a stereo camera.

The imaging section 10 is communicably connected to the teachingapparatus 105 via a cable. Wired communication using the cable isrealized by a standard such as Ethernet (registered trademark) or a USB.The imaging section 10 may be connected to the teaching apparatus 105 byusing wireless communication realized by a communication standard suchas Wi-Fi (registered trademark). The imaging section 10 and the teachingapparatus 105 may be formed not separately from each other butintegrally with each other. The imaging section 10 is provided at aposition where a range (hereinafter, referred to as an imaging range)including the work object O and capable of imaging the representativepoint OP of the work object O can be imaged. The imaging section 10captures a moving image as a captured image, but may alternativelycapture a still image, or a still image and a moving image as a capturedimage. The imaging section 10 is an example of an imaging device.

The robot 20 is a single-arm robot including, for example, the grippingportion HND, the manipulator MNP, and a plurality of actuator (notillustrated). The robot 20 may be a SCARA robot (horizontallyarticulated robot) or a dual-arm robot instead of the single-arm robot.The SCARA robot is a robot of which a manipulator is moved only in ahorizontal direction, and only a slide shaft at a front end of themanipulator is moved in a vertical direction. The dual-arm robot is arobot including two arms each of which is constituted by the grippingportion HND and the manipulator MNP.

The arm of the robot 20 is of a six-axis vertically articulated type,and can perform an operation of a six-axial degree of freedom throughinterlocking operations among the support stand, the manipulator MNP,and the gripping portion HND using the actuators. The arm of the robot20 may operate in five or less degrees of freedom (five axes) or mayoperate in seven or more degrees of freedom (seven axes). Hereinafter, adescription will be made of an operation of the robot 20, performed bythe arm including the gripping portion HND and the manipulator MNP. Thegripping portion HND of the robot 20 is provided with claws which cangrip an object.

The robot 20 is communicably connected to the control apparatus 130 via,for example, a cable. Wired communication using the cable is realized bya standard such as Ethernet (registered trademark) or a USB. The robot20 may be connected to the control apparatus 130 by using wirelesscommunication realized by a communication standard such as Wi-Fi(registered trademark). In the robot system 101, the robot 20 isconfigured to be connected to the control apparatus 130 which isprovided outside the robot 20 as illustrated in FIG. 13, but the controlapparatus 130 may be built into the robot 20 instead of thisconfiguration.

The robot 20 acquires a control signal generated by the controlapparatus 130, and moves the gripping portion HND and the manipulatorMNP of the robot 20 on the basis of the acquired control signal.

In response to a request from the teaching apparatus 105, the controlapparatus 130 outputs, to the teaching apparatus 105, informationindicating a control point position and attitude (that is, a controlpoint target position and attitude) obtained when a state is realizedwhich is desired by the user and is obtained right before the grippingportion HND grips the work object O through a user's jog operation.Then, the control apparatus 130 acquires reference information from theteaching apparatus 105.

When the robot system 101 causes the robot 20 to perform predeterminedwork, the control apparatus 130 causes the teaching apparatus 105 todetect a work object position and attitude on the basis of a capturedimage which is acquired by the teaching apparatus 105. The controlapparatus 130 acquires information indicating the work object positionand attitude detected by the teaching apparatus 105 as detectioninformation. The control apparatus 130 calculates a control point targetposition and attitude during the predetermined work on the basis of thereference information and the detection information.

The control apparatus 130 causes the robot 20 to move the grippingportion HND so that a control point position and attitude match thecalculated control point target position and attitude, and causes thegripping portion HND to grip the work object O so that a state (that is,relative position and attitude which are indicated by a reference offsetand are formed between the work object O and the gripping portion HND)of the same offset as the reference offset taught by the user isrealized. The control apparatus 130 controls the robot 20 so that thework object O gripped by the gripping portion HND is disposed at thearrangement position X.

A hardware configuration of the teaching apparatus 105 is the same asthe hardware configuration of the teaching apparatus 5 described withreference to FIG. 3 in the first embodiment, and thus descriptionthereof will be omitted here.

Next, with reference to FIG. 15, a functional configuration of theteaching apparatus 105 will be described. FIG. 15 is a diagramillustrating an example of a functional configuration of the teachingapparatus 105. The teaching apparatus 105 includes the storage section52, the input reception section 53, the communication section 54, thedisplay section 55, an image acquisition section 56, and a controlsection 160. Some or all function units included in the control section160 are realized by, for example, the CPU 51 executing the variousprograms stored in the storage section 52. In addition, some or all ofthe function units may be hardware function units such as a large scaleintegration (LSI) circuit or an application specific integrated circuit(ASIC).

The image acquisition section 56 acquires a captured image from theimaging section 10. The image acquisition section 56 outputs theacquired captured image to the control section 160.

The control section 160 includes a display control unit 161, a detectionunit 162, a computation unit 163, a communication control unit 166, andan imaging control unit 167. The control section 160 controls the entireteaching apparatus 105.

The display control unit 161 generates a GUI for detecting a work objectreference position and attitude on the basis of the acquired capturedimage, and displays the generated GUI. More specifically, the displaycontrol unit 161 generates a screen (GUI) for detecting the work objectreference position and attitude as a first teaching screen on the basisof a user's operation which is input by using the input receptionsection 53. The display control unit 161 controls the display section 55so as to display the generated first teaching screen. The displaycontrol unit 161 generates a screen for calculating a reference offsetas a second teaching screen. The display control unit 161 controls thedisplay section 55 to display the second teaching screen on the basis ofa user's operation which is input by using the input reception section53.

The detection unit 162 acquires a captured image obtained by the imagingsection 10 from the image acquisition section 56 on the basis of auser's operation which is input via the first teaching screen by usingthe input reception section 53. The detection unit 162 detects a workobject position and attitude on the basis of the acquired capturedimage. If the work object position and attitude is detected, thedetection unit 162 stores the detected work object position and attitudein the storage section 52 as a work object reference position andattitude.

The computation unit 163 includes a first calculation portion 164 and asecond calculation portion 165.

The first calculation portion 164 acquires a control point position andattitude from the control apparatus 130 as a control point targetposition and attitude via the communication section 54 controlled by thecommunication control unit 166 on the basis of a user's operation whichis input via the second teaching screen by using the input receptionsection 53. The first calculation portion 164 calculates a referenceoffset on the basis of the control point target position and attitudeacquired from the control apparatus 130 and the work object position andattitude detected by the detection unit 162. If the reference offset iscalculated, the first calculation portion 164 stores the calculatedreference offset in the storage section 52.

The second calculation portion 165 calculates a control point targetposition and attitude on the basis of the work object position andattitude detected by the detection unit 162 and the work objectreference position and attitude and the reference offset stored in thestorage section 52 in response to a request from the control apparatus130. The control section 160 may not include the second calculationportion 165.

The communication control unit 166 controls the communication section 54so that the control point position and attitude is acquired from thecontrol apparatus 130 on the basis of a user's operation which is inputby using the input reception section 53 via the second teaching screenwhich is displayed on the display section 55 by the display control unit161. The communication control unit 166 controls the communicationsection 54 so that information indicating the work object referenceposition and attitude detected by the detection unit 162 and informationindicating the reference offset calculated by the first calculationportion 164 are output to the control apparatus 130 on the basis of auser's operation which is input by using the input reception section 53via the second teaching screen which is displayed on the display section55 by the display control unit 161. The communication control unit 166causes the control point target position and attitude calculated by thesecond calculation portion 165 to be output to the control apparatus 130in response to a request from the control apparatus 130.

The imaging control unit 167 controls the imaging section 10 to capturean image of an imaging possible range.

Hereinafter, with reference to FIG. 16, a description will be made of aprocess in which the teaching apparatus 105 teaches the referenceinformation to the control apparatus 130. FIG. 16 is a flowchartillustrating an example of a flow of a process in which the teachingapparatus 105 teaches the reference information to the control apparatus130. First, the display control unit 161 displays a setting screen onthe basis of a user's operation which is input by using the inputreception section 53 (step S280). Next, the control section 160 of theteaching apparatus 105 receives a user's operation which is input viathe setting screen by using the input reception section 53 (step S290).Here, with reference to FIG. 17, the setting screen will be described.FIG. 17 is a diagram illustrating an example of the setting screen.

The display control unit 161 acquires a captured image obtained by theimaging section 10 from the image acquisition section 56, and displaysthe acquired captured image in a region V1. The display control unit 161displays a property column Q1 on the setting screen illustrated in FIG.17. The property column Q1 is a column which displays a list of itemswhich can be taught to the control apparatus 130 by the teachingapparatus 105 and is a column in which a name of each item and a setvalue of the item are displayed in correlation with each other.

For example, if the user selects the property RobotUOffset (an examplethe item) by using the input reception section 53, a button B10 isdisplayed in a set value column correlated therewith. When the userpresses (clicks on) the button B10 by using the input reception section53, the display control unit 161 displays the first teaching screendescribed in step S300.

The display control unit 161 displays a result column Q2 on the settingscreen illustrated in FIG. 17. The result column Q2 is a column whichdisplays various information pieces which have already been detected andare taught to the control apparatus 130 on the basis of the capturedimage obtained by the imaging section 10, and is a column which displaysa name of each item and a detection result for each item in correlationwith each other.

Hereinafter, a description will be made assuming that the user pressesthe button B10 illustrated in FIG. 17 by using the input receptionsection 53 in step S290. Next, the display control unit 161 generatesthe first teaching screen on the basis of the user's operation (that is,an operation of pressing the button B10) which is input by using theinput reception section 53 in step S290. The display control unit 161controls the display section 55 to display the generated first teachingscreen (step S300).

Next, the control section 160 of the teaching apparatus 105 receives auser's operation which is input via the first teaching screen by theinput reception section (step S310). Next, the first calculation portion164 detects a work object reference position and attitude on the basisof the received operation (step S320). Here, with reference to FIG. 18,a description will be made of the first teaching screen and processes insteps S300 to S320 related to the first teaching screen. FIG. 18 is adiagram illustrating an example of the first teaching screen.

If a first teaching screen UI1 illustrated in FIG. 18 is displayed onthe display section 55, the user causes the work object O to be placedin a range which can be imaged by the imaging section 10 according toprocedures shown by an explanation image E1 displayed on the firstteaching screen UI1. The imaging section 10 starts to capture an imageright after the first teaching screen UI1 is displayed, but mayalternatively start to capture an image on the basis of a user'soperation which is input via the first teaching screen UI1 by using theinput reception section 53.

On the first teaching screen UI1, in addition to the explanation imageE1, a captured image obtained by the imaging section 10 is displayed ina region P1. The display control unit 161 acquires the captured imageobtained by the imaging section 10 from the image acquisition section 56and displays the acquired captured image in the region P1. The user cancause the work object O to be placed in the range which can be imaged bythe imaging section 10 while checking the captured image displayed inthe region P1. The first teaching screen UT1 is an example of a firstscreen.

The captured image of the work object O which has already been placed inthe region which can be imaged by the imaging section 10 is displayed inthe region P1 illustrated in FIG. 18. After the user causes the workobject O to be placed in the region which can be imaged by the imagingsection 10, the user presses (clicks on) a teach button B11 illustratedin FIG. 18. If the user presses the teach button B11, the detection unit162 captures an image which is being displayed on the first teachingscreen UT1 when the teach button B11 is pressed, and detects a workobject position and attitude as a work object reference position andattitude on the basis of an image processing method such as patternmatching on the basis of the captured image.

For example, the detection unit 162 detects a position of the workobject O on the captured image and an attitude of the work object O onthe captured image by performing pattern matching on the basis of ashape of the work object O included in the captured image. Here, withreference to FIG. 19, a description will be made of a process ofdetecting the work object reference position and attitude, performed bythe detection unit 162. FIG. 19 is a diagram illustrating an example ofan image captured by the detection unit 162 on the basis of an operationwhich is input via the first teaching screen UI1 by using the inputreception section 53.

A captured image Pc1 is an example of an image captured by the detectionunit 162. In FIG. 19, directions on the captured image Pc1 arerepresented by an X axis and a Y axis illustrated in FIG. 19.Specifically, a horizontal direction of the captured image Pc1illustrated in FIG. 19 is set as an X direction, and a verticaldirection thereof is set as a Y direction. The detection unit 162detects a position of the representative point OP of the work object Oas a work object reference position from the captured image Pc1 throughpattern matching on the basis of the shape of the work object O.

The detection unit 162 detects, for example, a side OX and a side OYwhich extend from the representative point OP of the work object O, fromthe captured image Pc1 through the pattern matching on the basis of theshape of the work object O, and sets the extension direction of the sideOX as an X direction of the representative point OP and the extensiondirection of the side OY as a Y direction of the representative pointOP. The detection unit 162 detects the X direction and the Y directionof the representative point OP having the representative point OP as anorigin as a work object reference attitude. The detection unit 162stores the detected work object reference position and attitude in thestorage section 52.

Next, the display control unit 161 generates the second teaching screenon the basis of a user's operation which is input by using the inputreception section 53. The display control unit 161 controls the displaysection 55 to display the generated second teaching screen (step S330).Instead of the configuration in which the display control unit 161generates the second teaching screen on the basis of the user'soperation which is input by using the input reception section 53, thedisplay control unit 161 may generate the second teaching screen whenthe work object reference position and attitude are detected by thedetection unit 162 in step S320, or may generate the second teachingscreen with erasure of the first teaching screen as a trigger.

Next, the control section 160 of the teaching apparatus 105 receives auser's operation which is input via the second teaching screen by usingthe input reception section 53 (step S340). Next, the first calculationportion 164 acquires a control point target position and attitude fromthe control apparatus 130 on the basis of the received operation (stepS350). Next, the first calculation portion 164 reads the work objectreference position and attitude from the storage section 52, andcalculates a reference offset on the basis of the read work objectreference position and attitude and the control point target positionand attitude acquired in step S350 (step S360).

Here, with reference to FIG. 20, a description will be made of thesecond teaching screen and processes in steps S340 to S360 related tothe second teaching screen. FIG. 20 is a diagram illustrating an exampleof the second teaching screen. If a second teaching screen UI2illustrated in FIG. 20 is displayed on the display section 55, the usercauses the robot 20 to grip the work object O with the gripping portionHND according to procedures shown by an explanation image E2 displayedon the second teaching screen UI2. The user causes the robot 20 to gripthe work object O with the gripping portion HND through a jog operation.

The second teaching screen UI2 displays an explanation image E2 forshowing procedures performed when the control apparatus 130 is taughtvia the second teaching screen UI2. If the user follows the explanationshown by the explanation image E2, the user can easily perform anoperation for teaching the control apparatus 130 without reading amanual.

On the second teaching screen UI2, in addition to the explanation imageE2, a captured image obtained by the imaging section 10 is displayed ina region P2. The display control unit 161 acquires the captured imageobtained by the imaging section 10 from the image acquisition section 56and displays the acquired captured image in the region P2. If the secondteaching screen UI2 is displayed on the display section 55, the userpresses a teach button B12 which is displayed on the second teachingscreen UI2. Consequently, the user can perform an operation for teachingthe control apparatus 130 according to the procedures having the contentshown by the explanation image E2.

If the user presses the teach button B12, the display control unit 161displays a jog screen UI2-1 as a sub-screen of the second teachingscreen UI2 as illustrated in FIG. 21. FIG. 21 is a diagram illustratingan example of the second teaching screen UI2 and the jog screen UI2-1. Aplurality of buttons for moving the control point of the grippingportion HND of the robot 20 to a position desired by the user aredisplayed in a region JG on the jog screen UI2-1. For example, a +Xbutton is a button for moving the control point of the gripping portionHND in a positive direction of the X axis, and a −X button is a buttonfor moving the control point of the gripping portion HND in a negativedirection of the X axis.

In addition, a +Y button is a button for moving the control point of thegripping portion HND in a positive direction of the Y axis, and a −Ybutton is a button for moving the control point of the gripping portionHND in a negative direction of the Y axis. A +Z button is a button formoving the control point of the gripping portion HND in a positivedirection of the Z axis, and a −Z button is a button for moving thecontrol point of the gripping portion HND in a negative direction of theZ axis.

A +U button is a button for rotating the control point of the grippingportion HND in a positive direction about the X axis, and a −U button isa button for rotating the control point of the gripping portion HND in anegative direction about the X axis. A +V button is a button forrotating the control point of the gripping portion HND in a positivedirection about the Y axis, and a −V button is a button for rotating thecontrol point of the gripping portion HND in a negative direction aboutthe Y axis. A +W button is a button for rotating the control point ofthe gripping portion HND in a positive direction about the Z axis, and a−W button is a button for rotating the control point of the grippingportion HND in a negative direction about the Z axis.

The display control unit 161 displays, for example, the presentcoordinates or the like in a robot coordinate system of the controlpoint on the jog screen U12-1. The display control unit 161 displays thecaptured image obtained by the imaging section 10 in the region P2 ofthe second teaching screen UI2. The user can cause the work object O tobe gripped by the gripping portion HND through a jog operation via thejog screen UI2-1 while checking the captured image. A captured image ofa state in which the work object O is gripped by the gripping portionHND through the jog operation via the jog screen UI2-1 is displayed inthe region P2 of the second teaching screen UI2 illustrated in FIG. 21.The display control unit 161 may display a captured image in the regionP2 on the basis of an operation which is input via the first teachingscreen UI1 by using the input reception section 53. The second teachingscreen UI2 and the jog screen UI2-1 (that is, the screens illustrated inFIG. 21) are an example of a second screen.

After the user causes the work object O to be gripped by the grippingportion HND of the robot 20 through the jog operation via the jog screenUI2-1, the user presses (clicks on) a teach button B13 on the jog screenUI2-1 illustrated in FIG. 21. If the user presses the teach button B13,the first calculation portion 164 acquires information indicating acontrol point position and attitude obtained when the teach button B13is pressed, from the control apparatus 130 as information indicating acontrol point target position and attitude.

As mentioned above, in the teaching apparatus 105, the user causes thework object O to be actually gripped through a jog operation via the jogscreen U12-1, and acquires a control point position and attitude at aposition and an attitude where the work object O is gripped by thegripping portion HND, as a control point target position and attitude.Therefore, it is possible to acquire the control point target positionand attitude with higher accuracy than, for example, in a case where thecontrol point target position and attitude are detected on the basis ofa captured image. After the control point target position and attitudeare acquired from the control apparatus 130, the first calculationportion 164 reads a work object position and attitude from the storagesection 52 and calculates a reference offset on the basis of the readwork object reference position and attitude and the acquired controlpoint target position and attitude.

Here, with reference to FIG. 22, a description will be made of a processof acquiring the control point target position and attitude and aprocess of calculating the reference offset in the first calculationportion 164. FIG. 22 is a diagram illustrating an example of a capturedimage which is displayed in the region P2 of the second teaching screenU12 and is obtained when the work object O is gripped by the grippingportion HND through a jog operation.

A captured image Pc2 is an example of an image obtained when the workobject O is gripped by the gripping portion HND. In FIG. 22, directionson the captured image Pc2 are represented by an X axis and a Y axisillustrated in FIG. 22. Specifically, a horizontal direction of thecaptured image Pc2 illustrated in FIG. 22 is set as an X direction, anda vertical direction thereof is set as a Y direction. The firstcalculation portion 164 acquires a control point position and attitudeobtained in the state illustrated in FIG. 22 from the control apparatus130 as a control point target position and attitude.

Therefore, the teaching apparatus 105 can acquire the more highlyaccurate control point target position and attitude by using thecaptured image Pc2 than in a case of detecting the control point targetposition and attitude through pattern matching or the like. The firstcalculation portion 164 reads the work object reference position andattitude from the storage section 52, and calculates a reference offsetOS illustrated in FIG. 22 on the basis of the read work object referenceposition and attitude and the acquired control point target position andattitude.

Next, the communication control unit 166 controls the communicationsection 54 to output (teach) information indicating the work objectreference position and attitude detected in step S320 and informationindicating the reference offset calculated in step S360 to the controlapparatus 130 (step S370). Next, the display control unit 161 generatesa setting completion screen. The display control unit 161 controls thedisplay section 55 to display the generated setting completion screen(step S380).

Here, with reference to FIG. 23, the setting completion screen will bedescribed. FIG. 23 is a diagram illustrating an example of the settingcompletion screen. The display control unit 161 displays an image orexplanation for notifying that setting has been completed in a column E3on the setting completion screen. As illustrated in FIG. 23, the displaycontrol unit 161 may display a captured image P3 obtained by the imagingsection 10 on the setting completion screen but may not display thecaptured image. The display control unit 161 displays a completionbutton B15 on the setting completion screen. If the user presses thecompletion button B15, the display control unit 161 displays a settingscreen after teaching is finished, described in step S390.

Hereinafter, a description will be made assuming the user presses thecompletion button B15 on the setting completion screen. Next, thedisplay control unit 161 displays a setting screen again (step S390).Here, with reference to FIG. 24, a description will be made of a settingscreen after teaching is finished. FIG. 24 is a diagram illustrating anexample of the setting screen after teaching is finished.

The setting screen after teaching is finished, illustrated in FIG. 24displays not 0.000 but a value (for example, −174.220 pixels) of thereference offset calculated in step S320 as a set value OSR correlatedwith the property RobotUOffset unlike the setting screen before teachingis performed, illustrated in FIG. 17. As mentioned above, if teachingusing the first teaching screen UI1 and the second teaching screen U12is finished, the information taught to the control apparatus 130 isdisplayed on the setting screen.

As mentioned above, the teaching apparatus 105 teaches the referenceinformation to the control apparatus 130 through the processes in stepsS280 to S390.

After the reference information is taught to the control apparatus 130,the robot system 101 causes the robot 20 to perform predetermined work.At this time, the robot system 101 causes the imaging section 10 tocapture an image of the work object O every time a new work object O isplaced on the table TB, and detects a work object position and attitudeon the captured image on the basis of the captured image. The robotsystem 101 calculates a difference between the work object position andattitude and the work object reference position and attitude on thebasis of the detected work object position and attitude and the workobject reference position and attitude included in the referenceinformation which is taught by the 105.

The robot system 101 calculates a control point target position andattitude on the basis of the calculated difference (that is, a relativeposition and attitude of the work object O for the work object referenceposition and attitude) between the work object position and attitude andthe work object reference position and attitude and the reference offsetincluded in the reference information, and moves the gripping portionHND of the robot 20 so that the calculated control point target positionand attitude match the control point position and attitude. Then, therobot system 101 controls the robot 20 to grip the work object O and todispose the work object O at the arrangement position X. A differencebetween an attitude of the work object O and a reference attitude of thework object O is an example of an attitude of the work object.

In the robot system 101 according to the present embodiment, a referenceoffset is calculated by the computation unit 163, but, alternatively,the reference offset may be calculated by a separate device. In thiscase, the teaching apparatus 105 requests the device to calculate thereference offset via the control apparatus 130.

As described above, the robot system 101 according to the presentembodiment displays the first teaching screen UI1 for displaying thework object O placed within an imaging range of the imaging section 10and the second teaching screen UI2 for causing the robot 20 to grip thework object O placed within the imaging range of the imaging section 10,and detects a position and an attitude of the work object O by using aresult of the robot 20 gripping the work object O. Consequently, therobot system 101 can reduce a burden on the user related to teaching.

The robot system 101 detects the position and the attitude of the workobject O for the imaging range as a work object reference position andattitude on the basis of the work object O placed within the imagingrange of the imaging section 10, displayed on the first teaching screenUI1. Consequently, the robot system 101 can teach the detected workobject reference position and attitude to the control apparatus 130.

The robot system 101 computes a difference between a work objectposition and attitude of a new work object O placed within the imagingrange during predetermined work and the work object reference positionand attitude by using the work object reference position and attitude ofthe work object O detected by the detection unit 162 and a result of therobot 20 gripping the work object O. Consequently, the robot system 101can perform predetermined work on the work object O on the basis of areference attitude of the work object O, and the difference between thework object position and attitude and the work object reference positionand attitude, computed by using the result of the robot 20 gripping thework object O.

In the robot system 101, the second teaching screen UI2 includes theregion P2 in which a captured image obtained by the imaging section 10is displayed. Consequently, the robot system 101 can provide the userwith an environment in which the user moves the robot 20 through a jogoperation while checking the captured image.

In the robot system 101, the first teaching screen UI1 includes theregion in which work procedures are displayed and the second teachingscreen UI2 includes the region in which work procedures are displayed.Consequently, the robot system 101 reduces time and effort for the userto read a manual and thus allows the user to perform efficient work.

Third Embodiment

Hereinafter, a third embodiment of the invention will be described withreference to the drawings. In the present embodiment, the sameconstituent elements as in the first embodiment will be given the samereference numerals and will be described.

FIG. 25 is a diagram illustrating a configuration of a robot system 201according to the present embodiment. The robot system 201 includes animaging section 10, a single-arm robot 20 provided with a grippingportion HND (end effector) and a manipulator MNP, a control apparatus230, an image processing apparatus 40, and an information processingterminal 50. In the present embodiment, the single-arm robot indicates arobot having a single arm constituted by the gripping portion HND andthe manipulator MNP.

In the robot system 201, the imaging section 10 captures an image of arange including the work object O gripped by the robot 20. The workobject O is an object which can be gripped by the robot 20, and is anindustrial part such as a screw, a bolt, a gear, or a tool, but is notlimited thereto, and may be other objects as long as the objects can begripped by the robot 20. The work object O is an example of a workobject. The robot system 201 calculates relative positions and relativeattitudes (hereinafter, referred to as an offset) between a controlpoint TCP and a representative point OP which is set in the work objectO gripped by the robot 20 in advance on the basis of the captured imageof the range including the work object O.

The control point TCP indicates a position and an attitude of thegripping portion HND when the control apparatus 230 moves the grippingportion HND (and the manipulator MNP) of the robot 20, and indicates acentral point (a tool center point) of a flange provided at an end ofthe manipulator MNP at which the gripping portion HND is provided inthis example. The control point TCP is an example of a first position.The control point TCP is an example of a predetermined position of ahand. The representative point OP is a point indicating a position andan attitude of the work object O, set in the control apparatus 230 inadvance, and is a feature point of the work object O (in this example, acorner OP illustrated in FIG. 25). The representative point OP is anexample of a second position. The robot system 201 moves the controlpoint TCP by using the manipulator MNP and the gripping portion HND ofthe robot 20 so that a position and an attitude of the representativepoint OP (hereinafter, referred to as a work object position andattitude) match a position and attitude of the target point TP(hereinafter, referred to as a target point position and attitude) onthe basis of the calculated offset. Consequently, the robot system 201causes the robot 20 to dispose the work object O in the target area TAat an accurate position and attitude. Hereinafter, for convenience ofdescription, a position and an attitude of the control point TCP arereferred to as a control point position and attitude.

Here, the target point TP is a point in the target area TA in which thework object O is disposed by the robot 20 and is a point which matchesthe representative point OP when the work object O is disposed in thetarget area TA at an accurate position and attitude. The target area TAis an area in which the work object O is disposed by the robot 20, andis provided on a table TB in FIG. 25. The table TB is, for example, astand on which the robot 20 disposes the work object O.

The imaging section 10 is a camera including, for example, a chargecoupled device (CCD) or a complementary metal oxide semiconductor (CMOS)which is an imaging element converting collected light into an electricsignal. The imaging section 10 is a monocular camera, but may be formedof two or more cameras such as a stereo camera.

The imaging section 10 is communicably connected to the image processingapparatus 40 via a cable. Wired communication using the cable isrealized by a standard such as Ethernet (registered trademark) or auniversal serial bus (USB). The imaging section 10 may be connected tothe image processing apparatus 40 by using wireless communicationrealized by a communication standard such as Wi-Fi (registeredtrademark).

The imaging section 10 is provided at a position where a range(hereinafter, referred to as an imaging range) including theabove-described control point TCP and the representative point OP of thework object O gripped by the gripping portion HND can be imaged when therobot 20 moves the work object O to a predetermined imaging positionwith the manipulator MNP and the gripping portion HND. In FIG. 25, theimaging section 10 is provided at a position where an image of theabove-described imaging range can be captured vertically upward frombelow.

Instead of the configuration in which the imaging section 10 is providedat a position where an image of the above-described imaging range can becaptured vertically upward from below, there may be a configuration inwhich the imaging section 10 is provided at a position where an image ofthe imaging range can be captured in a horizontal direction, aconfiguration in which the imaging section 10 is provided at a positionwhere an image of the imaging range can be captured vertically downwardfrom above, and a configuration in which the imaging section 10 isprovided at a position where an image of the imaging range can becaptured from other directions. The imaging section 10 captures a stillimage as a captured image, but may alternatively capture a moving imageas a captured image. The imaging section 10 is an example of an imagingdevice.

The robot 20 is a single-arm robot including, for example, the grippingportion HND, the manipulator MNP, and a plurality of actuator (notillustrated). The robot 20 may be a SCARA robot (horizontallyarticulated robot) or a dual-arm robot instead of the single-arm robot.The SCARA robot is a robot of which a manipulator is moved only in ahorizontal direction, and only a slide shaft at a front end of themanipulator is moved in a vertical direction. The dual-arm robot is arobot including two arms each of which is constituted by the grippingportion HND and the manipulator MNP.

The arm of the robot 20 is of a six-axis vertically articulated type,and can perform an operation of a six-axial degree of freedom throughinterlocking operations among the support stand, the manipulator MNP,and the gripping portion HND using the actuators. The arm of the robot20 may operate in five or less degrees of freedom (five axes) or mayoperate in seven or more degrees of freedom (seven axes). Hereinafter, adescription will be made of an operation of the robot 20, performed bythe arm including the gripping portion HND and the manipulator MNP. Thegripping portion HND of the robot 20 is provided with claws which cangrip an object. The gripping portion HND is an example of a hand.

The robot 20 is communicably connected to the control apparatus 230 via,for example, a cable. Wired communication using the cable is realized bya standard such as Ethernet (registered trademark) or a USB. The robot20 may be connected to the control apparatus 230 by using wirelesscommunication realized by a communication standard such as Wi-Fi(registered trademark). In the robot system 201, the robot 20 isconfigured to be connected to the control apparatus 230 which isprovided outside the robot 20 as illustrated in FIG. 25, but the controlapparatus 230 may be built into the robot 20 instead of thisconfiguration.

The robot 20 acquires a control signal from the control apparatus 230,and moves the work object O gripped by the gripping portion HND of therobot 20 from the present position to a predetermined imaging positionwith the manipulator MNP on the basis of the acquired control signal.The robot 20 acquires a control signal from the control apparatus 230,and moves the work object O from the imaging position and disposes thework object O in the target area TA with the manipulator MNP on thebasis of the acquired control signal.

The control apparatus 230 outputs a control signal to the robot 20 so asto control the robot 20. The control apparatus 230 controls the robot 20to grip the work object O and to move the work object O to the imagingposition. The control apparatus 230 causes the image processingapparatus 40 to calculate the above-described offset. If informationindicating the offset is acquired from the image processing apparatus40, the control apparatus 230 calculates a control point position andattitude (hereinafter, referred to as a control point target positionand attitude) obtained when a representative point position and attituderelated to the work object O match a target point position and attitudeon the basis of the acquired information indicating the offset and thetarget point position and attitude which are registered in the controlapparatus 230 in advance via the information processing terminal 50. Thecontrol apparatus 230 calculates the control point target position andattitude and then causes the robot 20 to move the control point TCP sothat the control point position and attitude match the control pointtarget position and attitude.

In the robot system 201, instead of the configuration in which thecontrol apparatus 230 calculates the control point target position andattitude, the image processing apparatus 40 may calculate the controlpoint target position and attitude. In this case, the control apparatus230 outputs information indicating the target point position andattitude which are registered in the control apparatus 230 in advance,to the image processing apparatus 40. The image processing apparatus 40calculates the control point target position and attitude on the basisof the target point position and attitude acquired from the controlapparatus 230 and the offset calculated by the image processingapparatus 40, and outputs information indicating the calculated controlpoint target position and attitude to the control apparatus 230.

The image processing apparatus 40 causes the imaging section 10 tocapture an image of the above-described imaging range in response to arequest from the control apparatus 230. If a captured image is acquiredfrom the imaging section 10, the image processing apparatus 40calculates the above-described offset on the basis of the acquiredcaptured image. The image processing apparatus 40 outputs informationindicating the calculated offset to the control apparatus 230.

The information processing terminal 50 registers (inputs) variousinformation pieces in (to) the control apparatus 230. The informationprocessing terminal 50 is, for example, a notebook personal computer(PC), but may alternatively be a desktop PC, a tablet PC, a mobile phoneterminal, a multi-function mobile phone terminal (smartphone), apersonal digital assistant (PDA), or the like. The informationprocessing terminal 50 outputs information indicating theabove-described target point position and attitude to the imageprocessing apparatus 40 and registers (stores) the information therein.The information processing terminal 50 may be integrally formed with theimage processing apparatus 40.

Next, with reference to FIG. 26, a description will be made of ahardware configuration of the image processing apparatus 40. FIG. 26 isa diagram illustrating an example of a hardware configuration of theimage processing apparatus 40. The image processing apparatus 40includes, for example, a central processing unit (CPU) 41, a storagesection 42, an input reception section 43, and a communication section44, and performs communication with the control apparatus 230 via thecommunication section 44. These constituent elements are communicablyconnected to each other via a bus Bus. The CPU 41 executes variousprograms stored in the storage section 42.

The storage section 42 includes, for example, a hard disk drive (HDD), asolid state drive (SSD), an electrically erasable programmable read-onlymemory (EEPROM), a read only memory (ROM), and a random access memory(RAM), and stores various information pieces or images processed by theimage processing apparatus 40, and programs. Instead of being built intothe image processing apparatus 40, the storage section 42 may be anexternally attached storage device which is connected to a digital inputand output port such as a USB.

The input reception section 43 is, for example, a keyboard, a mouse, atouch panel, and other input devices. The input reception section 43 maybe configured of a touch panel so as to function as a display section.The input reception section 43 may not be provided in the imageprocessing apparatus 40. In this case, inputting to the image processingapparatus 40 may be received from either or both of the controlapparatus 230 and the information processing terminal 50.

The communication section 44 includes, for example, a digital input andoutput port such as a USB, or an Ethernet port.

Next, with reference to FIG. 27, a functional configuration of the imageprocessing apparatus 40 will be described. FIG. 27 is a diagramillustrating an example of a functional configuration of the imageprocessing apparatus 40. The image processing apparatus 40 includes thestorage section 42, the communication section 44, an image acquisitionsection 45, and a control section 46. Some or all function unitsincluded in the control section 46 are realized by, for example, the CPU41 executing the various programs stored in the storage section 42. Inaddition, some or all of the function units may be hardware functionunits such as a large scale integration (LSI) circuit or an applicationspecific integrated circuit (ASIC).

The image acquisition section 45 acquires a captured image from theimaging section 10. The image acquisition section 45 outputs theacquired captured image to the control section 46.

The control section 46 includes an imaging control unit 47, an offsetcalculation unit 48, and a communication control unit 49.

The imaging control unit 47 controls the imaging section 10 to capturean image.

The offset calculation unit 48 calculates an offset between a controlpoint position and attitude and a work object position and attitude onthe basis of a captured image acquired by the image acquisition section45.

The communication control unit 49 controls the communication section 44to output information indicating the offset calculated by the offsetcalculation unit 48 to the control apparatus 230.

Hereinafter, with reference to FIG. 28, a description will be made of aprocess in which the image processing apparatus 40 calculates an offset.FIG. 28 is a flowchart illustrating an example of a flow of a process inwhich the image processing apparatus 40 calculates an offset.Hereinafter, a description will be made of a process after the robot 20moves the work object O to an imaging position. First, the imagingcontrol unit 47 controls the imaging section 10 to capture an image ofthe above-described imaging range (step S400). Next, the imageacquisition section 45 acquires the captured image obtained in step S400from the imaging section 10, and outputs the acquired captured image tothe control section 46 (step S410).

Next, the offset calculation unit 48 calculates an offset between acontrol point position and attitude and a work object position andattitude on the basis of the captured image acquired from the imageacquisition section 45 (step S420). Here, with reference to FIGS. 29Aand 29B, a description will be made of a process of calculating theoffset between the control point position and attitude and the workobject position and attitude on the basis of the captured image. FIGS.29A and 29B show diagrams respectively illustrating an example of astate in which the imaging section 10 captures an image of the imagingrange and an example of a captured image obtained by the imaging section10.

FIG. 29A illustrates a state in which the imaging section 10 captures animage of the imaging range. In FIG. 29A, a description will be madeassuming that the work object O gripped by the gripping portion HND hasbeen moved to an imaging position by the robot 20. After the work objectO has been moved to the imaging position, the imaging section 10captures an image of the imaging range in a direction C illustrated inFIG. 29A. The direction C is a direction perpendicular to a face M ofthe work object O gripped by the gripping portion HND. The face M is aface which opposes the flange F at which the gripping portion HND isprovided when the work object O is gripped by the gripping portion HND.Therefore, the imaging section 10 captures an image of the imaging rangein a state in which the face M is parallel to a plane of the imagingelement.

The face M being made parallel to the plane of the imaging element isaimed at detecting an offset which is obtained when the work object O isdisposed in the target area TA at an accurate position and attitude andis an offset between a control point position and attitude and arepresentative point position and attitude on a target area face (thatis, on a plane of the table TB) when the control point TCP and therepresentative point OP are projected onto the target area face of thetarget area TA. As mentioned above, an offset is treated as an offset onthe target area face, and thus computation cost of the control apparatus230 or the image processing apparatus 40 can be minimized. Such offsettreatment is only an example, and other offset treatment methods may beused.

FIG. 29B illustrates an example of a captured image obtained by theimaging section 10. As illustrated in FIG. 29B, in the captured image,the face M is imaged from below. The captured image includes therepresentative point OP related to the work object O. The imageprocessing apparatus 40 causes the imaging section 10 to obtain acaptured image as illustrated in FIG. 29B and calculates an offset OSbetween a control point position and attitude and a work object positionand attitude on the basis of the captured image.

In FIG. 29B, the offset OS indicates a relative position and a relativeattitude on an XY plane which extend along the X axis and the Y axis onthe captured image. The image processing apparatus 40 detects, forexample, a shape of the gripping portion HND from the captured imagethrough pattern matching or the like so as to detect a position (thatis, a central position of the flange F) of the control point TCP on thecaptured image. The image processing apparatus 40 detects, for example,an attitude of the control point TCP on the captured image on the basisof a direction in which the manipulator MNP (not illustrated) includedin the captured image extends. When detecting an attitude of the controlpoint TCP, the image processing apparatus 40 may detect an attitude ofthe control point TCP by using other methods.

The image processing apparatus 40 detects a position on the capturedimage of the representative point OP related to the work object O, forexample, through pattern matching. The image processing apparatus 40detects a direction on the captured image of a side of the face M, forexample, through pattern matching, and detects an attitude of therepresentative point OP on the captured image on the basis of thedetected direction. When detecting the attitude of the representativepoint OP, the image processing apparatus 40 may detect the attitude ofthe representative point OP by using other methods. The image processingapparatus 40 calculates an offset OS in a robot coordinate system on thebasis of the detected control point position and attitude on thecaptured image and the work object position and attitude on the capturedimage. A coordinate (pixel coordinate) system on the captured image maybe correlated with the robot coordinate system through calibration.

Next, the communication control unit 49 controls the communicationsection 44 to output information indicating the offset between thecontrol point position and attitude and the work object position andattitude, calculated by the offset calculation unit 48 in step S420, tothe control apparatus 230 (step S430).

As mentioned above, the robot system 201 causes the image processingapparatus 40 which is formed separately from the control apparatus 230to perform an image process for calculating the offset between thecontrol point position and attitude and the work object position andattitude. Consequently, the robot system 201 can minimize a load such ascomputation related to the process performed by the control apparatus230. The robot system 201 can minimize the number of times ofcommunication performed by the control apparatus 230.

In the robot system 201, even if a new control apparatus X is providedwhich is different from the control apparatus 230 due to version-up ofthe control apparatus 230 which controls the robot 20, it is notnecessary to create a new program which has a format capable of beingexecuted by the control apparatus X and is related to an image processfor calculating an offset. In this case, in the robot system 201, ifcodes related to inputting and outputting of information with imageprocessing apparatus 40 have only to be written to a program of thecontrol apparatus X for controlling the robot 20, an offset can beeasily calculated, and the robot 20 can be controlled on the basis ofthe calculated offset.

As mentioned above, in the robot system 201 according to the presentembodiment, the image processing apparatus 40 calculates the offset OSbetween the control point position and attitude and the work objectposition and attitude by using a captured image obtained by the imagingsection 10. Consequently, the robot system 201 can efficiently controlthe robot 20.

The robot system 201 detects the control point position and attitudefrom the captured image. Consequently, the robot system 201 cancalculate the offset OS between the control point position and attitudeand the work object position and attitude.

The robot system 201 detects the offset OS between the control pointposition and attitude and the work object position and attitude from thecaptured image through the pattern matching. Consequently, the robotsystem 201 can detect the offset OS between the control point positionand attitude and the work object position and attitude from the capturedimage through the pattern matching.

As mentioned above, although the embodiments of the invention have beendescribed in detail with reference to the drawings, specificconfigurations are not limited to the embodiments and may be changed,replaced, omitted, or the like without departing from the spirit of theinvention.

A program for realizing the function of any function unit in theabove-described apparatus (for example, the teaching apparatus 5 of therobot system 1) may be recorded on a computer readable recording medium,and the program may be read to a computer system and be executed. The“computer system” mentioned here is assumed to include an operatingsystem (OS) or hardware such as a peripheral apparatus. The “computerreadable recording medium” includes a portable medium such as a flexibledisk, a magneto-optical disk, a read only memory (ROM), or a compactdisc (CD)-ROM, and a hard disk built into the computer system. Inaddition, the “computer readable recording medium” is assumed to includea memory which holds a program for a certain period of time, such as avolatile memory (random access memory (RAM)) of a computer systemserving as a server or a client when the program is transmitted via anetwork such as the Internet or a communication line such as a telephoneline.

The program may be transmitted from a computer system in which theprogram is stored in a storage device or the like to another computersystem via a transmission media or by using a carrier wave in thetransmission medium. Here, the “transmission medium” for transmittingthe program includes a medium having a function of transmittinginformation, such as a network (communication network) such as theInternet or a communication line such as a telephone line.

The program may realize some of the above-described functions. Theprogram may be a so-called differential file (differential program)which can realize the above-described functions in combination with aprogram which has already been recorded in the computer system.

What is claimed is:
 1. A teaching apparatus for a robot configured tomove a work object, the teaching apparatus comprising: a memoryconfigured to store computer-readable instructions; a display configuredto display the work object and teaching instructions for the robot; ascreen configured to display operating commands of the robot; an inputdevice configured to allow the screen to accept the operating commands;and a processor configured to execute the computer-readable instructionsso as to: cause an imaging device to capture an image of the work objectthat is placed in an image capturing region, the imaging device beingconfigured to output the captured image; cause the screen to accept theoperating commands via the input device so as to operate a hand of therobot such that the hand grasps the work object; and calculate aposition and attitude of the work object based on a position andattitude of the hand when the hand grasps the work object.
 2. Theteaching apparatus according to claim 1, wherein the processor isconfigured to define the position and attitude of the work object as areference position and attitude of the work object with respect to theimage capturing region when the work object is placed in the imagecapturing region and is displayed in the screen.
 3. The teachingapparatus according to claim 1, wherein the work object is configuredwith first and second work objects different from each other, whereinthe processor is configured to define a position and attitude of thefirst work object as a reference position and attitude of the first workobject with respect to the image capturing region when the first workobject is placed in the image capturing region and is displayed on thescreen, and the processor is configured to calculate a position andattitude of the second work object that is placed in the image capturingregion by using the position and attitude of the hand when the handgrasps the first work object and the reference position and attitude ofthe first work object.
 4. The teaching apparatus according to claim 1,wherein the screen includes a region in which the captured imageobtained by the imaging device is displayed.
 5. The teaching apparatusaccording to claim 1, wherein either or both of the display and thescreen include a region in which a work procedure is displayed.
 6. Arobot system comprising: a robot having a hand, the robot beingconfigured to move a work object; and a teaching apparatus configured toteach a position and attitude of the work object to the robot, theteaching apparatus includes: a memory configured to storecomputer-readable instructions; a display configured to display the workobject and teaching instructions for the robot; a screen configured todisplay operating commands of the robot; an input device configured toallow the screen to accept the operating commands; and a processorconfigured to execute the computer-readable instructions so as to: causean imaging device to capture an image of the work object that is placedin an image capturing region, the imaging device configured to outputthe captured image; cause the screen to accept the operating commandsvia the input device so as to operate the hand of the robot such thatthe hand grasps the work object; and calculate the position and attitudeof the work object based on a position and attitude of the hand when thehand grasps the work object.
 7. The robot system according to claim 6,wherein the processor is configured to define the position and attitudeof the work object as a reference position and attitude of the workobject with respect to the image capturing region when the work objectis placed in the image capturing region and is displayed in the screen.8. The robot system according to claim 6, wherein the work object isconfigured with first and second work objects different from each other,wherein the processor is configured to define a position and attitude ofthe first work object as a reference position and attitude of the firstwork object with respect to the image capturing region when the firstwork object is placed in the image capturing region and is displayed onthe screen, and the processor is configured to calculate a position andattitude of the second work object that is placed in the image capturingregion by using the position and attitude of the hand when the handgrasps the first work object and the reference position and attitude ofthe first work object.
 9. The robot system according to claim 6, whereinthe screen includes a region in which the captured image obtained by theimaging device is displayed.
 10. The robot system according to claim 6,wherein either or both of the display and the screen include a region inwhich a work procedure is displayed.